-- Enter time in localised time zones.
   Currently all dates in RestfulDB are entered in GMT (see #559 for detailed discussion). It would be more convenient to the user to enter dates in localised (system) time zones. (RM#929, https://projects.ibt.lt/repositories/issues/929)

-- Make the default database reader user configurable.
   As of r7172 the database reader user name is @reader@. However, some databases have different public user names and forcing them all to add @reader@ alias might be deemed too expensive. Therefore I suggest making this configurable in metadata (after #710 is implemented). The metadatabase would be still accessed using @reader@, but for accessing the content database RestfulDB would use the public user name found in the metadata.

I need this feature to implement #896, as the public user for the *OD databases is @cod_reader@. Providing read access for @reader@ is another option, but I believe the described feature is worth implementing on its own. (RM#897, https://projects.ibt.lt/repositories/issues/897)

-- Tables that are composed views take too long to load by RestfulDB.
   Tables that are composed views take too long to load. See the related issue for reproducing example.

The problem seems to be in RestfulDB layer; during all the request time, the server runs 'db.pl' and 'mysql' at max loads.

The 'mysql' CLI client query, however, runs in about 1 sec. (RM#848, https://projects.ibt.lt/repositories/issues/848)

-- Do not suggest a new identifier when editing an entry.
   The issue manifests at least since r7116.

The @"numbers"@ metadata table can be used to configure RestfulDB to suggest values for certain columns (i.e. SolsaID). When properly configured, the suggested value and an additional "Insert" button appears near the column field. This is useful when creating new entries, however, it is quite misleading when editing entries that already have previously assigned column values (see issue #777). Possible approaches when dealing with entries that already have a previously assigned column value:

1) Do not suggest a new value nor generate the "Insert" button;
2) Suggest a new value, generate the button, but change the button text from "Insert" to "Replace".

Similar behaviour is produced when dealing with dates and timestamps, however, it is unclear if it should be handled in the same way. (RM#840, https://projects.ibt.lt/repositories/issues/840)

-- RestfulDB queries return inconsistent JSON results.
   As noted by Fabio:

the answer at the following query:

https://solsa.crystallography.net/db-trunk/test_samples/sample/ER-MK00-0020?depth=2&amp;format=json

is returning an inconsistency in the answer, specifically in the part related to the "Related Sample"

As you can notice the primary sample id is sometimes returned as an int and other times as a  data structure.

We should make it so that one type only (int or structure) is returned.


This is the part I'm interested in:

"related_sample": {
"data": [
{
"attributes": {
"id": 217,
"primary_sample_id": 64,
"related_sample_id": {
"attributes": {
"SolsaID": "ER-MK00-0012",
"acquisition_date": null,
"id": 422,
"kind": "natural",
"name": "DI1 013",
"preparation_conditions": null,
"revision_id": 1543,
"special_details": null,
"specimen_type": "1/2 core",
"uuid": "3d0a74ca-305e-11e9-9910-3b60e34a9608"
},
"id": 422,
"type": "sample"
},
"revision_id": {
"attributes": {
"db_user": "root@localhost",
"env_text": null,
"http_host": null,
"http_user_agent": null,
"id": 2682,
"log_message": "Switched primary and related samples for multiple entries as specified by Aisha Kanzari.",
"remote_addr": null,
"remote_user": "antanas@localhost",
"server_addr": null,
"server_name": null,
"timestamp": "2020-07-30 10:52:13",
"uuid": "946fb943-d239-11ea-b8c7-525400d148ab"
},
"id": 2682,
"type": "revision"
},
"special_details": null,
"uuid": "4241c772-305e-11e9-9910-3b60e34a9608"
},
"id": 217,
"links": {
"self": "https://solsa.crystallography.net/db-trunk/test_samples/related_sample/217"
},
"type": "related_sample"
},
{
"attributes": {
"id": 269,
"primary_sample_id": {
"attributes": {
"SolsaID": "ER-MK00-0014",
"acquisition_date": null,
"id": 445,
"kind": "natural",
"name": "DN004",
"preparation_conditions": null,
"revision_id": 1554,
"special_details": null,
"specimen_type": "polished section",
"uuid": "4ad86886-342e-11e9-9e00-f573e34a9608"
},
"id": 445,
"type": "sample"
},
"related_sample_id": 64,
"revision_id": {
"attributes": {
"db_user": "Lydia Ikama",
"env_text": "AUTH_TYPE: Basic\nCONTENT_LENGTH: 33455\nCONTENT_TYPE: multipart/form-data; boundary=---------------------------23986132909161\nCONTEXT_DOCUMENT_ROOT: /var/www/html\nCONTEXT_PREFIX: \nDOCUMENT_ROOT: /var/www/html\nGATEWAY_INTERFACE: CGI/1.1\nHTTPS: on\nHTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\nHTTP_ACCEPT_ENCODING: gzip, deflate, br\nHTTP_ACCEPT_LANGUAGE: en-US,en;q=0.5\nHTTP_CONNECTION: keep-alive\nHTTP_DNT: 1\nHTTP_HOST: solsa.crystallography.net\nHTTP_REFERER: https://solsa.crystallography.net/db/samples/sample/new\nHTTP_UPGRADE_INSECURE_REQUESTS: 1\nHTTP_USER_AGENT: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0\nPATH: \nQUERY_STRING: \nREDIRECT_HTTPS: on\nREDIRECT_SSL_TLS_SNI: solsa.crystallography.net\nREDIRECT_STATUS: 200\nREDIRECT_URL: /db/samples/sample/new\nREMOTE_ADDR: 85.115.60.202\nREMOTE_PORT: 34856\nREMOTE_USER: Lydia Ikama\nREQUEST_METHOD: POST\nREQUEST_SCHEME: https\nREQUEST_URI: /db/samples/sample/new\nSCRIPT_FILENAME: /var/www/html/db/cgi-bin/new.pl\nSCRIPT_NAME: /db/cgi-bin/new.pl\nSERVER_ADDR: 172.16.1.101\nSERVER_ADMIN: webmaster@localhost\nSERVER_NAME: solsa.crystallography.net\nSERVER_PORT: 443\nSERVER_PROTOCOL: HTTP/1.1\nSERVER_SIGNATURE: &lt;address&gt;Apache/2.4.10 (Debian) Server at solsa.crystallography.net Port 443&lt;/address&gt;\n\nSERVER_SOFTWARE: Apache/2.4.10 (Debian)\nSSL_TLS_SNI: solsa.crystallography.net",
"http_host": "solsa.crystallography.net",
"http_user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0",
"id": 1554,
"log_message": null,
"remote_addr": "85.115.60.202",
"remote_user": "Lydia Ikama",
"server_addr": "172.16.1.101",
"server_name": "solsa.crystallography.net",
"timestamp": "2019-02-19 10:08:24",
"uuid": "4ad82920-342e-11e9-9e00-f573e34a9608"
},
"id": 1554,
"type": "revision"
},
"special_details": null,
"uuid": "52080c6a-342e-11e9-9e00-f573e34a9608"
},
"id": 269,
"links": {
"self": "https://solsa.crystallography.net/db-trunk/test_samples/related_sample/269"
},
"type": "related_sample"
},
{
"attributes": {
"id": 273,
"primary_sample_id": {
"attributes": {
"SolsaID": "ER-MK00-0015",
"acquisition_date": null,
"id": 446,
"kind": "natural",
"name": "DI1 013",
"preparation_conditions": null,
"revision_id": 1555,
"special_details": null,
"specimen_type": "polished section",
"uuid": "b81eb95e-342e-11e9-bf32-3c74e34a9608"
},
"id": 446,
"type": "sample"
},
"related_sample_id": 64,
"revision_id": {
"attributes": {
"db_user": "Lydia Ikama",
"env_text": "AUTH_TYPE: Basic\nCONTENT_LENGTH: 33618\nCONTENT_TYPE: multipart/form-data; boundary=---------------------------23655155744031\nCONTEXT_DOCUMENT_ROOT: /var/www/html\nCONTEXT_PREFIX: \nDOCUMENT_ROOT: /var/www/html\nGATEWAY_INTERFACE: CGI/1.1\nHTTPS: on\nHTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\nHTTP_ACCEPT_ENCODING: gzip, deflate, br\nHTTP_ACCEPT_LANGUAGE: en-US,en;q=0.5\nHTTP_CONNECTION: keep-alive\nHTTP_DNT: 1\nHTTP_HOST: solsa.crystallography.net\nHTTP_REFERER: https://solsa.crystallography.net/db/samples/sample/new\nHTTP_UPGRADE_INSECURE_REQUESTS: 1\nHTTP_USER_AGENT: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0\nPATH: \nQUERY_STRING: \nREDIRECT_HTTPS: on\nREDIRECT_SSL_TLS_SNI: solsa.crystallography.net\nREDIRECT_STATUS: 200\nREDIRECT_URL: /db/samples/sample/new\nREMOTE_ADDR: 85.115.60.202\nREMOTE_PORT: 62260\nREMOTE_USER: Lydia Ikama\nREQUEST_METHOD: POST\nREQUEST_SCHEME: https\nREQUEST_URI: /db/samples/sample/new\nSCRIPT_FILENAME: /var/www/html/db/cgi-bin/new.pl\nSCRIPT_NAME: /db/cgi-bin/new.pl\nSERVER_ADDR: 172.16.1.101\nSERVER_ADMIN: webmaster@localhost\nSERVER_NAME: solsa.crystallography.net\nSERVER_PORT: 443\nSERVER_PROTOCOL: HTTP/1.1\nSERVER_SIGNATURE: &lt;address&gt;Apache/2.4.10 (Debian) Server at solsa.crystallography.net Port 443&lt;/address&gt;\n\nSERVER_SOFTWARE: Apache/2.4.10 (Debian)\nSSL_TLS_SNI: solsa.crystallography.net",
"http_host": "solsa.crystallography.net",
"http_user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0",
"id": 1555,
"log_message": null,
"remote_addr": "85.115.60.202",
"remote_user": "Lydia Ikama",
"server_addr": "172.16.1.101",
"server_name": "solsa.crystallography.net",
"timestamp": "2019-02-19 10:11:27",
"uuid": "b81e6bd4-342e-11e9-bf32-3c74e34a9608"
},
"id": 1555,
"type": "revision"
},
"special_details": null,
"uuid": "bf904e96-342e-11e9-bf32-3c74e34a9608"
},
"id": 273,
"links": {
"self": "https://solsa.crystallography.net/db-trunk/test_samples/related_sample/273"
},
"type": "related_sample"
}
]
} (RM#811, https://projects.ibt.lt/repositories/issues/811)

-- Add means to specify an optional revision message when creating, editing or deleting data.
   The issue manifests at least since r7073.

The 'revision' table [1] in the 'samples' database has the 'log_message' column which is intended to contain a human-readable description of the changes that were carried out in the given revision. Currently, this column is only used by database migration and data management scripts, however, it would be useful to make it accessible to regular user as well. The addition of this option would allow database users to convey their intentions in a much clearer way since currently even the smallest change in a top level parent table (i.e. the 'samples' table) changes the revision numbers of all related entries in other tables (i.e. experiment descriptions).

This option could be implemented by adding a generic text field to the top of the entry creation, deletion and editing dialogues.

[1] https://solsa.crystallography.net/db-trunk/test_samples/revision (RM#783, https://projects.ibt.lt/repositories/issues/783)

-- An extra '?' is added to the URL when using the Prev/Next buttons in a record card.
   The issue manifests at least since r7073.

Steps to reproduce:
1. Go to any card page wit no additional GET parameters (i.e. [1]);
2. Press the Next button.

The button redirects to the page of the next entry, however, the URL contains an extra '?' symbol, i.e. "https://solsa.crystallography.net/db-trunk/test_samples/sample/5?".



[1] https://solsa.crystallography.net/db-trunk/test_samples/sample/3 (RM#782, https://projects.ibt.lt/repositories/issues/782)

-- Asynchronous record update using Ajax.
   The idea is to upload part of the record in 'modify.pl' script and make the page load faster. If the number of related records exceed the 'rows' variable, the pagination should be used. (RM#772, https://projects.ibt.lt/repositories/issues/772)

-- Number of found Related items.
   It's useful to show the number of found &lt;Related&gt; items next to related table name and link. Without loading full page. (RM#764, https://projects.ibt.lt/repositories/issues/764)

-- Unauthorised curl -X PUT to experiment_fusion table.
   Problem using the new certificate to PUT the attached file as a row in the experiment_fusion table:
 
curl -k -sS -d @C\test_json_\0717_TestID2A_SW_flow\test1_experiment_fusion_ER-NC00-0056_parameters.json --header "Content-Type:application/json" -K C:\test_json_\credentials.pem --pass ***** -X PUT https://solsa.crystallography.net/db/test_samples/experiment_fusion

returns : 
This server could not verify that you"            
  "are authorized to access the document"               
  "requested.  Either you supplied the wrong"           
  "credentials (e.g., bad password), or your"           
  "browser doesn't understand how to supply"            
  "the credentials required.&lt;/p&gt;"

A similar command used to work with the former certificate. For now none of the certificates work. I noticed it 2 weeks ago. (RM#761, https://projects.ibt.lt/repositories/issues/761)

-- User's SSL certificate obstructs otherwise anonymous read access.
   I tried accessing a MySQL database. The browser (Firefox) prompted to supply user's SSL certificate, which I did. However, the access was denied to me on the server side:
	&lt;pre&gt;
	Cannot connect: Access denied for user 'Andrius Merkys'@'localhost' to database '&lt;db&gt;'
	&lt;/pre&gt;
I learned that there is passwordless anonymous MySQL user account ('reader') for everyone not having SSL certificates. Thus I had to hide my certificate in order to access the database, which is quite inconvenient.

Maybe the code could fallback to 'reader' account should the SSL-provided credentials fail?

Observed on a server running RestfulDB trunk of r6980. (RM#752, https://projects.ibt.lt/repositories/issues/752)

-- Spreadsheet data is not inserted in a single transaction.
   @saulius has reported that after timed-out spreadsheet uploads the SQL database state is changed (some of the records are inserted). This means that either rollback is not performed correctly, or data is inserted in separate transactions. It has been agreed that RestfulDB should insert the whole spreadsheet in a single transaction.

I got the impression that the problem was spotted using SQLite3 backend. (RM#751, https://projects.ibt.lt/repositories/issues/751)

-- Related tables should be collapsed by default.
   It has been suggested that related tables in single record previews should be collapsed by default. (RM#741, https://projects.ibt.lt/repositories/issues/741)

-- Input fields allowing one to navigate to specific page/record.
   Input fields have been requested to navigate to specific page/record in table/single record previews, correspondingly. Currently, there are captions "Page X of N" and "Record X of N". Instead of "X" there should be an input field allowing to easily navigate to specific page/record. (RM#740, https://projects.ibt.lt/repositories/issues/740)

-- OOM kill in table listing page.
   This issue has been detected while testing RestfulDB on COD validation messages database. RestfulDB trunk of r6950 was used.

It seems that @Database::get_fks_cell_text()@ is executing @SELECT ... FROM &lt;table_a&gt; INNER_JOIN &lt;table_b&gt; WHERE &lt;FK description&gt;@ and then processing every row with Perl code. For large tables this operation requires a lot of resources. On 2020-06-22 @alex said he knows how to resolve this issue with AJAX, thus I am assigning this issue to him. (RM#729, https://projects.ibt.lt/repositories/issues/729)

-- XLSX tab name is limited by 31 characters.
   Somehow, XLSX tab name is again limited by 31 characters and this should happen only to XLS, not XLSX. In order to replicate the bug, try to type:
	&lt;pre&gt;
	https://solsa.crystallography.net/db-trunk/test_samples/rock_name?format=xlsx
	&lt;/pre&gt;

It will produce:
	&lt;pre&gt;
	Sheetname rock_structure_description_history must be &lt;= 31 chars at /var/www/web-apps/restfuldb-versions/trunk/lib/Spreadsheet.pm line 694.
	&lt;/pre&gt;

What is strange, the same happens not only to 'db-trunk', but also to 'db'.

Noticed that new spreadsheet data structures are involved, because the error originates from:
	&lt;pre&gt;
	$spreadsheet-&gt;add_worksheet( $table );
	&lt;/pre&gt; (RM#711, https://projects.ibt.lt/repositories/issues/711)

-- Refactor database information (i.e. database name) into a separate table.
   The issue manifests at least since revision 6894.

At least up to schema version 0.2.1, the database name is provided in each row of the column definition using the 'description'.'dbname' column. What is more, the database name column exists in several tables ('description', 'fk_format'). This is highly denormalised and makes data management much more difficult.

Information related to the described database should be refactored into a separate table. The new table could be called 'db_instance', 'database_instance' or something similar (table name 'database' would probably clash with DBMS keywords).

Having a separate database information table would provide the following benefits:
1. Simplify the RestfulDB dump loading logic. Currently, the dump loading script has to keep track of the dbname columns which need to be updated; 
2. Allow to provide additional details about the database (i.e. if it should be displayed in the listing of all databases);
3. Allow to avoid long indices in the 'description' table. This would provided greater compatibility with different MariaDB versions and other database management systems. For example, even after reducing the length of the 'dbname' and 'dbcolumn' columns, issue #611 still manifests under older MariaDB version (i.e. 10.1.44-MariaDB, the default version under Debian 9); 
4. All other benefits of having a (more) normalised database schema and adhering to the "DRY principle":https://en.wikipedia.org/wiki/Don%27t_repeat_yourself (see issue #632). (RM#710, https://projects.ibt.lt/repositories/issues/710)

-- MySQL issues an "Illegal mix of collations" error when uploading UUID values with non-ASCII symbols.
   The issue manifests at least since revision 6894.

Steps to reproduce:
1. Go to the https://solsa.crystallography.net/db-trunk/test_samples/colour_chroma
2. Download a template file with a single row;
3. Fill in the required data in the file;
4. Change the UUID value to contain a non-ASCII symbol, i.e. ('ž');
5. Try to upload the file.

Instead of detecting the incorrect value due to the UUID regular expression, the following MySQL error message is issued:
	&lt;pre&gt;
	'SELECT `colour_chroma`.* FROM `colour_chroma` WHERE ((`chroma` = ?) AND (`uuid` = ?))' failed: Illegal mix of collations (ascii_general_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '=' at /var/www/web-apps/restfuldb-versions/trunk/lib/Database.pm line 2275.
	&lt;/pre&gt;

Uploading an UUID value that contains an illegal ASCII symbols (i.e. 'G') results in an error message related to the regular expressions:
	&lt;pre&gt;
	value for 'uuid' ('g06de32c-a339-11ea-82ea-2951bcc593e0') does not match regular expression '^[a-f0-9]{8}-(?:[a-f0-9]{4}-){3}[a-f0-9]{12}$', cannot insert invalid record
	&lt;/pre&gt; (RM#709, https://projects.ibt.lt/repositories/issues/709)

-- Add parent record ID columns to spreadsheet templates.
   In today's videoconference @darius expressed the need to have parent ID columns (&lt;parent_table&gt;.&lt;id_column&gt;) in spreadsheet templates for children records. Currently all unique parent columns (except ID columns) are added in spreadsheet templates.

Pros:
* A user will not need to add parent ID columns oneself.

Cons:
* Number of columns in spreadsheet template sheet will increase by at most the number of parent tables. (RM#707, https://projects.ibt.lt/repositories/issues/707)

--  Display measurement units next to column names in spreadsheets.
   Implement #631 in spreadsheets (RM#681, https://projects.ibt.lt/repositories/issues/681)

-- Record caching may result in cache inconsistency due to changed ID column.
   (RM#670, https://projects.ibt.lt/repositories/issues/670)

-- Implement the same download formats (CSV,ODS,XLS,etc.) for downloading individual records as for tables.
   This feature would be nice since it would make the RestfulDB system more orthogonal.

Currently a workaround exists: one can search using filters for one particular record and download data in spreadsheet format for this selection.

It is awkward however when the '&amp;format=json' QS parameter works for both tables and records, and '&amp;format=csv' – only for tables. (RM#665, https://projects.ibt.lt/repositories/issues/665)

-- Implement image thumbnails.
   Implement a special RestfulDB column Role for storing image thumbnails (a small version of the images stored in the same table). Thumbnails should use 'fk_target' to specify for which image they are generated. Thumbnails can be generated for both images stored in BLOBs (role/coltype "image") and image links (role/coltype "imguri").

In the list representation of a table, if thumbnails are present, they SHOULD be used instead of original (large) images. If all images in the table have corresponding thumbnail fields, then the table should revert to returning larger number of rows per page (50 or 100, not 10; see issue #662). In the record view, however, the image version itself SHOULD be displayed and available for download.

The enlarged image, however, SHOULD preferably display and/or download the original (large) image, if that is easy to implement without JS. (RM#663, https://projects.ibt.lt/repositories/issues/663)

-- Tables with (large) images should return 10 rows by default, not 100; otherwise loading takes very long time, especially over slow links.

Images are considered "large" if they are not thumbnails.

The presence of images SHOULD be detected from the column RestfulDB Role. (RM#662, https://projects.ibt.lt/repositories/issues/662)

-- Add "increase" and "decrease" control elements next to the "Rows in data file:" and "Rows in template file:" entry fields (on the right side), so that they can be also operated using mouse, not only keyboard. (RM#661, https://projects.ibt.lt/repositories/issues/661)

-- Optimise space for user name and colophon.
   Currently (as of v0.14.2) on the RestfulDB Web pages authenticated user name takes the whole line with wide vertical padding; the "colophon" (the version and grant number indication below) takes two lines, but contains a lot of unused space.

It is suggested to reduce this space by moving user name elsewhere (maybe to the top line, next to the database name), and leaving just one line for versions and grant numbers. The "back to top" and "breadcrumb" links do not seem to be useful and can be removed altogether. Actually, the "you are here" "breadcrumb" link duplicates database link information available on the top of the page. The information on the top of the page is more prominent and more likely to be used. (RM#659, https://projects.ibt.lt/repositories/issues/659)

-- Decrease padding around values in table cells.
   In both the "list" and the "card" display forms of database records padding is very wide around values, resulting in fewer values per page as would be needed for comfortable data inspection. The suggestion is to reduced vertical padding space to admit more values or more records on one page. (RM#658, https://projects.ibt.lt/repositories/issues/658)

-- Should we represent measurement units in JSONs or in JSON schemas?.
   Measurement units will be picked from XML descriptions of the databases and displayed on the Web pages, entry forms and spreadsheets. Should the same information be presented in machine-readable descriptions of the data, such as JSON/JSON schemas and XML/XML schemas, so that REST API users are aware of the measurement units that MUST be used? (RM#657, https://projects.ibt.lt/repositories/issues/657)

-- Provide intermediate Web form to choose tables for editing when "Edit" button for a VIEW is selected.
   While DB views can not be edited, the end user or data manager may well have a need to update data reflected in the VIEW. A general solution for updating VIEW values is, IMHO, unfeasible (e.g. there is no easy self-evident way to update computed values); however individual tables from which the VIEW is constructed can be edited as usual.

To help data managers to find these tables, it would be very nice if the "Edit" (or even the "New") button from a VIEW display page would lead to a selection HTML page, explaining that the "table you have selected for editing is a computed view and can not be edited directly. Instead, you can edit the underlying database tables:", and providing the links to those tables.

The SQL statement parser in this cases only needs to extract the list of tables from which the VIEW is created, which seems much easier and better defined task than figuring out which VIEW fields can be updated directly. (RM#656, https://projects.ibt.lt/repositories/issues/656)

-- Move sorting buttons-arrows to the right of the text in Web tables.
   Currently, a lot of vertical space in Web pages is wasted since it is empty. One way to make presentation more compact would be to:

# stack sorting arrows in column headings in top of each other, not next to each other;
# moving these arrows in the GUI to the right of the (possibly folded) column names.

This would make the whole line free for displaying data. Of course the sorting arrows MUST NOT be randomly moved to the position below the title.

No that we have alternative column names implemented and column names folded nicely, and column names have adjustable width, this feature should work nicely. (RM#647, https://projects.ibt.lt/repositories/issues/647)

++ Show VIEW names (as opposed to TABLE names) and their content in italics.
   It would be beneficial for a Web user to have clear unobtrusive indication which tables and which values are computed (and thus not editable directly). I suggest using italic font for displaying VIEW names, possibly VIEW column names and VIEW values. (RM#646, https://projects.ibt.lt/repositories/issues/646)

-- Regular expressions on numeric fields are not applied at JS level.
   The issue manifest at least since revision 6689.

Steps to reproduce:
1. Create a table with a numeric field;
2. Add a validation regular expression on the numeric field (i.e. [^9]);
3. Enter value '9' into the field. Note, that JS does not notify about the regex violation;
4. Press the "Save" button.

Upon trying to save the value an error message of the following form is generated:
	&lt;pre&gt;
	value for '&lt;column_name&gt;' ('9') does not match regular expression '^[^9]$', cannot insert invalid record
	&lt;/pre&gt; (RM#641, https://projects.ibt.lt/repositories/issues/641)

-- Provide the means to specify numeric value ranges via metadata.
   The issue manifest at least since revision 6689.

Currently, there is no straightforward way of limiting numeric values to a certain range (i.e. [0;1]). A similar result can be achieved by using regular expressions, however, this approach is very inconvenient both for the developer and the user. For example, regular expression '([01]([.]0*)?)|(0?[.][0-9]+)' limits values to the [0;1] range, however, this is not immediately clear upon reading the regular expression and may be completely incomprehensible for the regular user. I propose adding a new metadata table which would record value limit information in a more convenient way.

Possible table structure:

	&lt;pre&gt;
	CREATE TABLE IF NOT EXISTS `value_range` (
	  `id` INT(11) NOT NULL,
	  `column_id` INT(11) NOT NULL,
	  `min_value` FLOAT NULL,
	  `is_min_inclusive` TINYINT(1) NULL,
	  `max_value` FLOAT NULL,
	  `is_max_inclusive` TINYINT(1) NULL
	);
	&lt;/pre&gt;

Note, that similar mechanism of specifying value ranges already exist in JSON Schema (https://json-schema.org/understanding-json-schema/reference/numeric.html#range). (RM#640, https://projects.ibt.lt/repositories/issues/640)

-- Updating an entry with a field marked as 'file' or 'image' generates a warning.
   The issue manifests at least since revision 6681.

Steps to reproduce:
1. Go to a table with a field marked as 'file' or 'image' in the metadata (i.e. https://solsa.crystallography.net/db-trunk/test_samples/file );
2. Create a new entry in the table;
3. Try to edit the newly created entry.

The following message should be generated upon the update:

	&lt;pre&gt;
	Cannot insert file contents of field '&lt;field_name&gt;': either file cannot be read or not a file submitted. at /var/www/html/restful/lib/Database.pm line 641.
	&lt;/pre&gt; (RM#634, https://projects.ibt.lt/repositories/issues/634)

-- The data type of 'dbname' and 'dbcolumn' columns in table 'fk_format' should be same as in table 'description'.
   The issue manifests at least since revision 6667.

In release 0.2.0 of the schema the type of 'dbname' and 'dbcolumn' columns in the 'description' table was changed from '@VARCHAR(255)@' to '@VARCHAR(128)@'. However, semantically similar 'dbname' and 'dbcolumn' columns in the 'fk_format' table were left unchanged. The column type of columns in the 'fk_format' table should also be reduced to '@VARCHAR(128)@' to keep things consistent. (RM#632, https://projects.ibt.lt/repositories/issues/632)

-+ Display measurement units next to column names.
   A consensus was reached that measurement units are required to be shown next to column names in HTML and spreadsheet formats. There was a suggestion to use a separate metatable to do so (for SPOT reasons). (RM#631, https://projects.ibt.lt/repositories/issues/631)

-- Display alternative column names in search forms.
   Original SQL column names are displayed in the search forms as of now. I suppose it would be much more useful to show alternative column names (aka altnames) there. (RM#629, https://projects.ibt.lt/repositories/issues/629)

-- HTML ids are not unique in entry pages with related tables.
   The issue manifests at least since revision 6663.

Steps to reproduce:
1. Go to an entry with related tables (i.e. https://solsa.crystallography.net/db-trunk/test_samples_migration/sample/5);
2. View the source code.

The 'table-constrainer' id is reused for each div of the related tables. This does not seem to cause immediate usability errors. (RM#628, https://projects.ibt.lt/repositories/issues/628)

-- Display alternative column names in spreadsheets.
   Currently, spreadsheets give the column names as they are provided in the SQL tables. The consensus was reached to replace them with alternative column names as they were introduced in #515 (with additional units). (RM#617, https://projects.ibt.lt/repositories/issues/617)

+- Uploading a CSV+ZIP file as CSV causes a MySQL error.
   The issue manifests at least since revision 6600.

Steps to reproduce the issue will be provided using the 'samples' database:

1. Go to the 'colour_chroma' table;
2. Download an empty CSV template (set row number to 0);
3. Select the CSV upload format and upload the downloaded CSV+ZIP template file.

The upload action causes the following server error:
	&lt;pre&gt;
	You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1 at /var/www/html/restful/lib/Database.pm line 2797.
	&lt;/pre&gt; (RM#615, https://projects.ibt.lt/repositories/issues/615)

-- Data from an XLSX file upload have corrupt characters.
   As reported by @Darius, symbol sequences '-&gt;' are converted to "strange" characters. (RM#590, https://projects.ibt.lt/repositories/issues/590)

-- There is no distinct URI to present a new form.
   Now that the action of the RestfulDB Web site is determined by the HTTP request type (GET/POST/PATCH/...), generating a for for entering a new record (e.g. when pressing the "New" button) actually uses the same URI (e.g. https://solsa.crystallography.net/db/samples/sample), and the functionality is "hidden" in the request type.

This breaks the main REST assumption (functionality is represented by an end-point, state is represented by a payload); this makes it difficult (impossible) to make links that point to new record forms.

A suggestion would be to use QS parameter to request a new record form; now that we already have the "action=..." QS parameter, the following extension seems natural:

https://solsa.crystallography.net/db/samples/sample?action=new

This can be implemented along the existing functionality with little added code, and without breaking the existing system compatibility. We need to think about that. (RM#575, https://projects.ibt.lt/repositories/issues/575)

-- Error pages (e.g. HTTP 500 page) should report RestfulDB version; possibly also the latest DB Schema version.

OS name, version, HTTP server name would be also helpful.

We need to think about security. Possibly, this information should only be provided to authenticated users. (RM#574, https://projects.ibt.lt/repositories/issues/574)

-- Tables that have more than one extkey column do not allow to access individual entries .
   The issue manifests at least since revision 6441.

Trying to access a specific entry via the website when clicking on a value of an extkey column fails if the table contains several extkey columns with overlapping values.

Steps to reproduce the issue will be described using the 'samples' database.

Steps to reproduce:
1. Go to the 'compound' table;
2. Create a new entry. Fill in both the 'ec_number' and the 'cas_number' columns;
3. Try to enter a specific entry by clicking on the 'ec_number' value.

The user is redirected to a website with the following error message:

	&lt;pre&gt;
	table 'compound' has more than one external key column: 'cas_number', 'ec_number', cannot continue due to ambiguity
	&lt;/pre&gt;

The issue could probably be resolved by generating website links with the appropriate value of the 'id_column=' parameter. (RM#568, https://projects.ibt.lt/repositories/issues/568)

++ Metadata from the 'resolver' table is applied even if the 'description.coltype' does not require it.
   The issue manifests at least since revision 6441.

Description of the 'resolver' table states that:
	&lt;pre&gt;
	-- For the 'resource' and possibly for the 'uuid' and 'extkey'
	-- columns, the `resolver` table can contain base URLs that can be
	-- used to convert resource values (IDs) to Web URI references. Simple
	-- concatenation is supposed to be used:
	&lt;/pre&gt;
However, the URL link is always constructed regardless of the value of the 'description.coltype' column (including the @data@ type which should suppress any interpretation of the column). Of course, this might be a feature and not a bug, but then the description in the metatables.sql file should be updated accordingly. (RM#567, https://projects.ibt.lt/repositories/issues/567)

++ Date and time suggestions use GMT although mention localised time zones.
   All captions for date and time suggestions use GMT dates and times, although mentions localised time zones:

	&lt;pre&gt;
	  &lt;td&gt;
	    &lt;label class="mobile-cell"&gt;ctime&lt;/label&gt;
	    &lt;span class="cell-breaker"&gt;&lt;br/&gt;&lt;/span&gt;
	    &lt;input id="input:data:0.ctime" name="column:data:0.ctime" type="text" value=""/&gt;
	    Current server time (EET): 07:06:42
	    &lt;input type="hidden" name="" value="07:06:42" id="select:data:0.ctime"/&gt;
	    &lt;input type="button" name="insert_default" value="Insert" onclick="insert_suggested_value('data:0.ctime', '');" class="button bordered shadowed small responsive-padding responsive-margin" id="default:data:0.ctime" style="display:none"/&gt;
	    &lt;script type="text/javascript"&gt;document.getElementById('default:data:0.ctime').style.display='inline'&lt;/script&gt;
	  &lt;/td&gt;
	&lt;/pre&gt;
Tested on 09:06:42 EET time.

I consider this to be a bug. It would be better to inform the user that GMT dates and times are shown. The reason why this issue has not manifested before might be that the server is using GMT/UTC as its default time zone. (RM#559, https://projects.ibt.lt/repositories/issues/559)

-- Apache2 authentication directive 'Satisfy' is deprecated.
   According to https://httpd.apache.org/docs/trunk/upgrading.html, 'Satisfy' directive is deprecated, therefore should be replaced. (RM#557, https://projects.ibt.lt/repositories/issues/557)

-- Data should be read in a single transaction.
   Saulius has suggested that all the data in a single response should be read wrapped in a transaction. That means, all the required reads should be immune to intermediate modifications of the data. (RM#552, https://projects.ibt.lt/repositories/issues/552)

-+ Generate JSON schema for a database.
   Saulius and others have requested automatic generation of JSON schema for live databases. There should be a specific request instruction (for example, 'action=schema') to download a schema corresponding both to the current SQL and metadatabase instructions. (RM#551, https://projects.ibt.lt/repositories/issues/551)

-- Need an authentication endpoint.
   After discussion with Fabio (fabio.brigidi@gmail.com 2020-03-04 11:13 to grazulis@ibt.lt) and the Vilnius team, a decisions is taken to create special authentication endpoints:

a request to:

https://solsa.crystallography.net/authenticate

MUST return "204 No Content" if the user is authenticated during the access the server endpoints (e.g. for https://solsa.crystallography.net/db/samples), and MUST return "401 Unauthorized" if there is no authentication record. Authorisation is further dealt with DB GRANT layer, or with extra authorisations on specifics endpoints that MAY return "403 Forbidden" if specific endpoints (e.g. https://solsa.crystallography.net/db/samples_restful) require extra authorisation.

Most probably, the easiest way to implement it will be .htaccess rules. (RM#545, https://projects.ibt.lt/repositories/issues/545)

-- Solsa DB requests take long time to complete.
   All requests, including GET, and generation of new record entry forms, take too long to complete (15-30 seconds on the current Solsa server).

This is too long for interactive work. Also, for JSON GET requests it starts hurting since these are supposed to be used, in larger numbers, in interactive GUI software. The code needs to be profiled and bottlenecks need to be fixed.

From the server load (top) inspections, it seems that 'mysql' process takes most (75%) of the time in long-response cases.

A possible way to fix: use only one query for a "layer" of returned tables to get all records in one query, and not separate SQL queries for each row.

Needs some investigation. (RM#544, https://projects.ibt.lt/repositories/issues/544)

-- Error pages provide the same link to HTTP status 500 for all error statuses.
   The issue manifests at least since revision 6251.

The error page contains a link of the following form:

	&lt;pre&gt;
	More information on HTTP status XXX
	&lt;/pre&gt;

where XXX is the HTTP code, i.e. 403.

Despite of the error code the link always points to the description of the 500 error code (https://tools.ietf.org/html/rfc7231#section-6.6.1). (RM#541, https://projects.ibt.lt/repositories/issues/541)

-- Trying to upload a file that the user does not have proper permission to read results in a server error.
   The issue manifests at least since revision 6241.

Steps to reproduce:
1. Create a non-empty file which is not readable to the user (i.e. root:root, 700);
2. Try to upload the file to any file table.

This results in an Internal Server Error. (RM#540, https://projects.ibt.lt/repositories/issues/540)

-- Updating a record having an image results in "either file cannot be read or not a file submitted".
   The full warning message follows:

&gt; Cannot insert file contents of field 'image': either file cannot be read or not a file submitted. at /var/www/html/restful/lib/Database.pm line 626. 

No harm seems to be done. (RM#538, https://projects.ibt.lt/repositories/issues/538)

-- Reduce the number of SQL SELECT in get_record_descriptions() by querying tables.
   Currently, get_record_descriptions() fetches the records of top-level table and adds its parent and children records by issuing SELECT queries one-by-one. This could be optimised a lot by combining these SELECT queries together. (RM#526, https://projects.ibt.lt/repositories/issues/526)

-- Possible JavaScript injections and XSS attacks in Markdown fields.
   While all other fields' values are HTML-escaped in generated HTML pages, Markdown-formatted fields' values are placed verbatim as received from Markdown -&gt; HTML translator. Thus there is a possibility to inject JavaScript via plain HTML by using onClick or similar actions. A possible solution would be to strip HTML tags from Markdown-formatted values prior to passing them to HTML translator.

Marking this as a low priority task, as the possible attacker must have write privilege to Markdown fields. Nevertheless this should be solved, as world-writable databases should be still made safe to visit. (RM#520, https://projects.ibt.lt/repositories/issues/520)

-- Client-side validation of dates is not robust enough.
   Some date values are still treated as valid even though they cannot be inserted into the database, i.e. 2019-11-31. However, proper date validation might be a bit too complex to implement if no date-handling libraries are available. (RM#514, https://projects.ibt.lt/repositories/issues/514)

-+ Multiple clicks on the 'save' button result in multiple data inserts.
   This is bug has a tendency of manifesting when bad Internet connection and impatient users are combined. Multiple clicks on the same save button results in multiple instances of the same values being inserted into the database.

One of the possible solutions for this problems would be a JS script that captures the clicks or disables the button. (RM#512, https://projects.ibt.lt/repositories/issues/512)

-- Rename: db.pl -&gt; table.pl; modify.pl -&gt; record.pl.
   Current names of CGI scripts do not reflect their intended purpose. Thus I suggest renaming:

* db.pl -&gt; table.pl;
* modify.pl -&gt; record.pl

And, possibly:

* tables.pl -&gt; db.pl. (RM#511, https://projects.ibt.lt/repositories/issues/511)

-- Special symbols in filenames are not properly escaped when displayed in the 'content' column.
   The issue manifests at least since revision 6051.

Prerequisite -- a properly configured file table:
1) The table should contain at least columns 'filename' and 'content';
2) The 'filename' and 'content' columns should have the following descriptions in the metadata database:
2.1) 'content' should have the 'file' coltype;
2.2) 'filename' should have the 'filename' coltype and point to the 'content' column.

The example will be provided using the 'file' table from the 'samples' database.

Steps to reproduce:
1) Create a file with special symbols, i.e. "&lt;test.txt";
2) Insert file into the file table.

The '&lt;' symbol will be properly encoded as a HTML entity in the 'filename' column, but will result in broken HTML in the 'content' column. (RM#510, https://projects.ibt.lt/repositories/issues/510)

-- File upload via spreadsheet is working improperly.
   Currently, spreadsheet columns containing file contents are stored in the database as is, additionally issuing a warning:

    Cannot insert file contents of field 'content': either file cannot be read or not a file submitted. at /var/www/html/restful/lib/Database.pm line 432. 

Looking at the code it seems that file columns are not treated differently from other data columns in Spreadsheet.pm, and this has to be fixed. However, the following issue has to be addressed first: do we treat contents of the spreadsheet columns as proper file contents, or hyperlinks from where the contents should be loaded? (RM#509, https://projects.ibt.lt/repositories/issues/509)

-- Use limit for related tables in single record preview.
   Currently, all related records are shown. At least 'rows' parameter should be handled (with an appropriate warnings that values for the tables were trimmed). (RM#502, https://projects.ibt.lt/repositories/issues/502)

-- ON DUPLICATE KEY UPDATE bumps revisions upon inserting the same data.
   Noted when uploading the same CSV file over and over again. It would be nice to avoid bumping revisions, but it's just a minor issue. (RM#496, https://projects.ibt.lt/repositories/issues/496)

-- Foreign keys are ordered by their IDs even if they are shown as formatted text.
   Ordering should probably be based on the string representation ("FK format") shown to the end user. However, this is difficult to achieve, as ordering is done at DBMS level, and FK representation is the responsibility of RestfulDB layer. (RM#495, https://projects.ibt.lt/repositories/issues/495)

-- Configurable per-table default ordering.
   (RM#494, https://projects.ibt.lt/repositories/issues/494)

-- Validate the sign of the number as requested by SQL schema.
   Currently the type and length of submitted data are checked prior to insertion/update in the database tables. However, number sign check is not done, mostly due to the fact that there is no reliable way to get it by inspecting live MariaDB/SQLite database. (RM#488, https://projects.ibt.lt/repositories/issues/488)

-- Enforce snake_case in query string parameter names.
   Name of 'idColumn' should be changed to 'id_column'. Others should also be reviewed in an attempt to unify the naming convention. (RM#476, https://projects.ibt.lt/repositories/issues/476)

-- Change the name and capitalisation of the project to RestfulDB.
   Various captions, titles and texts refer to the project as 'restful', 'Restful', 'RESTfulDB', 'RESTful DB' and others. They all should be changed to use 'RestfulDB'. Perl packages already use correct name. (RM#475, https://projects.ibt.lt/repositories/issues/475)

-- Remove or make configurable SOLSA-specific captions to make the project more customisable.
   Currently (as of r5903) all page titles and footers contain string 'SOLSA Databases', which is the case for Open Samples Database, but not for every other database. As RestfulDB is supposed to be a generic tool for manipulating SQL databases, I suggest either removing these SOLSA references altogether, or make them customisable by some means, for example, configurable via metadatabase.

Of course, a reference to SOLSA must remain in the footer to acknowledge the funding source. However, this has to be written in explicit language. (RM#473, https://projects.ibt.lt/repositories/issues/473)

-- Add an advanced search field to enter filter string.
   Saulius has proposed to add an advanced search field where an user could type in (or paste in) a filter string to be searched. Currently filters could be specified by manipulating the query string only. (RM#472, https://projects.ibt.lt/repositories/issues/472)

-- Limit returned related tables in JSON format output via query string parameter.
   There is a need to limit the returned JSON data by including only user-requested tables. Similar objective is achieved in JSON API by using @inlude@ request parameter: https://jsonapi.org/format/#fetching-includes. (RM#470, https://projects.ibt.lt/repositories/issues/470)

-- Generate SolsaID and other automatically generatable fields.
   (RM#468, https://projects.ibt.lt/repositories/issues/468)

-- Hide duplicated FKs to the same table in data entry HTML forms.
   For example, 'related_sample' table has two FKs to 'sample', thus the related table is shown twice. We should make it appear only once. (RM#466, https://projects.ibt.lt/repositories/issues/466)

-- Changing the ordering does not reset to the first page.
   I cannot imagine a use case when a user would want to stay on the same page after changing the ordering. (RM#459, https://projects.ibt.lt/repositories/issues/459)

-- Add warning for "dangling tables".
   If a subordinate table is filled, but not the upper level table, no data are inserted and no warning is given. A warning should be provided as a feedback to the user. (RM#446, https://projects.ibt.lt/repositories/issues/446)

-- 'Save as new' functionality behaves strangely with records that have attached files.
   The issue manifests at least since revision 5831.

Steps to reproduce:
1. Begin editing a record that has an attached file (i.e. one from the 'samples'.'xrd_image' table);
2. Change any of the fields, but do not modify the file.
3. Press the 'Save as new' button.

The newly created entry will have the NULL value instead of the file contents. (RM#443, https://projects.ibt.lt/repositories/issues/443)

-- Outdated descriptions in the *_restful.description table should be handled as warnings rather than errors.
   The issue manifests at least since revision 5829.

RestfulDB reacts very harshly to inconsistencies in the description table. Consider the following situation:

1) Table A references table B;
2) Table A is dropped;
3) Trying to view entries in table B results in a fatal error. (RM#442, https://projects.ibt.lt/repositories/issues/442)

-- Links with '#related_expandable' no longer work.
   Hrefs to '#related_expandable' no longer work, albeit they are generated in modify.pl. (RM#428, https://projects.ibt.lt/repositories/issues/428)

-- Tables without ID columns fail to be accessed by external keys.
   Spotted in a view table by Antanas. (RM#424, https://projects.ibt.lt/repositories/issues/424)

-- Encrypted columns are made visible by using them in views.
   This sounds like mitigation of private data. (RM#422, https://projects.ibt.lt/repositories/issues/422)

-- Default ordering does not seem to work in spreadsheet downloads.
   Observed in downloads in 'revision' table using curl. (RM#421, https://projects.ibt.lt/repositories/issues/421)

-- Record ID may be 0.
   Therefore, the Perl check `if( $record_id ) { ... }` may have unexpected consequences. (RM#419, https://projects.ibt.lt/repositories/issues/419)

-- Spreadsheet downloads do not handle 'columns' parameter.
   Instead of limited set of columns, all columns of a table are returned. Not 100% sure whether this is a bug. (RM#403, https://projects.ibt.lt/repositories/issues/403)

-- Header button 'Selection' is missing in the page shown right after inserting new records.
   (RM#402, https://projects.ibt.lt/repositories/issues/402)

-- Parameter 'columns' is not transferred to db.pl by clicking the header button.
   Header button for database does not transfer 'columns' setting to db.pl. Not 100% sure whether this is a bug or a feature. (RM#401, https://projects.ibt.lt/repositories/issues/401)

-- Warnings/errors are needed when adding child records to parents not mentioned in spreadsheets.
   Two situations should be reported as warnings/errors in:

Relative URL: ^/trunk/tests/cases/db.pl_235.sh
Repository Root: svn+ssh://saulius-grazulis.lt/home/saulius/svn-repositories/restful
Repository UUID: e4f2a99c-8986-4a80-949c-bc855d164b74
Revision: 5755 (RM#395, https://projects.ibt.lt/repositories/issues/395)

-- Spreadsheet::csv_to_record_descriptions() is N*N when it should be NlogN.
   (RM#393, https://projects.ibt.lt/repositories/issues/393)

-- No check on compatibility between spreadsheet data and table during the upload.
   Simple check of top table name is needed, because now it is allowed to go to 'sample' table
and try to upload 'experiment_xrd'. The upload just silently ignores all tables. There should
be warning or error. (RM#391, https://projects.ibt.lt/repositories/issues/391)

-- Quiet ignore upsert on wrong not UNIQUE foreign key values in spreadsheet upload.
   When uploading through spreadsheets, foreign keys are ignored if the values are incorrect, but not UNIQUE. It should give an error. (RM#383, https://projects.ibt.lt/repositories/issues/383)

-- Mark mandatory columns in ODS, XLS and XLSX spreadsheet templates.
   (RM#380, https://projects.ibt.lt/repositories/issues/380)

-- Multiple FKs from the same table result in duplicated compound column names.
   In data/template spreadsheet for table 'related_sample', both 'primary_sample_id' and 'related_sample_id' would be transformed to 'sample.id'. This both looks weird, and may be ambiguous both to human operator, both to the processing software. A clear solution has to be found in how to deal with such issues. One of them is to fallback to the original column name, if possible. However, in spreadsheet templates for 'related_sample', both columns might be named 'sample.uuid', and would not have an underlying column in table 'related_sample'. (RM#368, https://projects.ibt.lt/repositories/issues/368)

-- Identify parent records by URLs in spreadsheet downloads and uploads.
   There has been a discussion that parent records in spreadsheet downloads and uploads should be identified via URLs instead of ID values. However, IDs of parent records currently are not shown at all (#707). It should be decided how both this issue and #707 should be resolved. (RM#362, https://projects.ibt.lt/repositories/issues/362)

-- Check undefined value and empty line distinction in spreadsheet formats.
   (RM#360, https://projects.ibt.lt/repositories/issues/360)

-- New lines that are entered using the website are displayed with additional '_x000D_' sequences in the downloaded XLSX files.
   The issue manifests in revision 5665.

Steps to reproduce:
1. Go to any table that has a text field;
2. Enter a value with a new line;
3. Download the data as an XLSX file.

Upon opening the file it can be seen that an additional '_x000D_' text sequence is placed near the new lines symbol.

Interestingly enough, the same symbol is not visible in the multi line values that were uploaded directly via SQL upload statements. (RM#354, https://projects.ibt.lt/repositories/issues/354)

-- Strange interaction between the filter and the upload form.
   Issue manifests in the 'algirdas-spreadsheet-upload-GUI' branch at least since revision 5638.

Steps to reproduce:
1) Enter any table;
2) Do not enter any search parameters, but still press the 'Filter' button;
3) Select any upload format and any file (even of the incorrect type) and press 'Upload'.

Instead of trying to upload the data, the website redirects to a new filtered page with the filter parameter (Page 0 of 0 selected with (id = "")). (RM#353, https://projects.ibt.lt/repositories/issues/353)

-- The download template allows to specify an unreasonably high number of template lines.
   Steps to reproduce:
1. Go to any table that offers a template download;
2. Enter '1000000' as the number of rows in template;
3. Initiate the download of template in XLS form.

The XLS form takes a long time to generate, but eventually finishes. However, only 65536 UUID values are generated in each sheet (probably a limitation of XLS).

Since this behaviour might be used as an attack vector, the number of template lines should be restricted to a reasonable number (i.e. 100 or a 1000). (RM#350, https://projects.ibt.lt/repositories/issues/350)

-- Newline symbols get converted to '&amp;#10;' when inserting multiline values via the XLSX upload.
   Issue manifests in the 'algirdas-spreadsheet-upload-GUI' branch at least since revision 5638. (RM#349, https://projects.ibt.lt/repositories/issues/349)

-- Provide radio buttons to select between INSERT, UPDATE and UPSERT in spreadsheet uploads.
   Currently, UPSERT is a silent default, whether it should not be. Nevertheless, there should be an option in the HTML form to select it. (RM#345, https://projects.ibt.lt/repositories/issues/345)

-- Nonexistent tables and columns have to be reported in spreadsheet uploads.
   Currently they seem to be plainly ignored. (RM#344, https://projects.ibt.lt/repositories/issues/344)

-- Search for '\' (single backslash) does not match backslashes in data.
   (RM#337, https://projects.ibt.lt/repositories/issues/337)

-- Child records for already existing parents are silently ignored in spreadsheet uploads.
   As demonstrated by tests/cases/db.pl_235.sh. Could be possibly fixed by using Database::modify_record_descriptions(). (RM#327, https://projects.ibt.lt/repositories/issues/327)

-- Child records for non-existing parents are silently ignored in spreadsheet uploads.
   As demonstrated by test case tests/cases/db.pl_235.sh. (RM#326, https://projects.ibt.lt/repositories/issues/326)

-- Add comment to records in metatable 'validation_regex'.
   Currently metatable 'validation_regex' contains only 'id' and 'regex' columns. A human-readable comment column would be beneficial, as interpreting regular expressions alone could be tedious. (RM#322, https://projects.ibt.lt/repositories/issues/322)

-- Generate SolsaID and other automatically generatable fields in spreadsheet templates.
   Currently there is a mechanism to suggest values for external keys a la SolsaID. However, spreadsheet templates do not support them as of now.

Possible caveat: this seriously increases the risk to overwrite previously inserted records during UPSERT. (RM#320, https://projects.ibt.lt/repositories/issues/320)

-- Investigate value-based addressing in ODS, XLS and XLSX spreadsheets.
   In multisheet spreadsheets it would be beneficial to have value-linked columns to represent foreign keys. (RM#315, https://projects.ibt.lt/repositories/issues/315)

-- Files of unrecognised format cannot be downloaded.
   It was reported that uploaded Windows Media Files could neither be viewed nor downloaded. (RM#309, https://projects.ibt.lt/repositories/issues/309)

-- Implement internal codes for errors, warnings and notices.
   Internal error codes should be implemented for errors, warnings and notices. (RM#307, https://projects.ibt.lt/repositories/issues/307)

-- Return warnings and notes in JSON format responses.
   Currently, only errors are returned for JSON format responses. Warnings and notes should be returned also. (RM#306, https://projects.ibt.lt/repositories/issues/306)

-- Forbid lowercase HTTP requests.
   According to https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.1, "[t]he method is case-sensitive". Thus we should respond with 'Unknown method' to 'get', 'delete' and the like. (RM#303, https://projects.ibt.lt/repositories/issues/303)

-- Distinguish deletion of existing record from deletion of non-existing record.
   The second HTTP submission with '-X DELETE' returns JSON with empty ID:

	&lt;pre&gt;
	saulius@koala doc/ $ curl --cacert ~/tmp/SOLSA/JSON/uploads/all.pem -sSL https://solsa.crystallography.net/db/test_samples/sample?filter=solsaid+CONTAINS+%22SG%22\&amp;format=json\&amp;depth=0 | jq -r '.data[] | (.id|tostring) + "\t" + (.attributes | (.SolsaID + "\t" + .special_details + "\t" + .name))'
	562	VU-TEST-SG01	test sample	test upload via JSON POST
	577	VU-TEST-SG02	test sample (3)	test update via JSON POST, with the SolsaID specified
	
	saulius@koala uploads/ $ gpg --quiet --batch --decrypt credentials.cfg.gpg.asc | sed 's|certificates/CA/ca-chain.crt|all.pem|' | curl -K - -X DELETE -sSL --fail https://solsa.crystallography.net/db/test_samples/sample/VU-TEST-SG02\?format=json 
	{"meta":{"detail":"Record '577' was deleted."}}
	
	saulius@koala uploads/ $ gpg --quiet --batch --decrypt credentials.cfg.gpg.asc | sed 's|certificates/CA/ca-chain.crt|all.pem|' | curl -K - -X DELETE -sSL --fail https://solsa.crystallography.net/db/test_samples/sample/VU-TEST-SG02\?format=json 
	{"meta":{"detail":"Record '' was deleted."}}
	&lt;/pre&gt;

&lt;s&gt;The empty "Record '' was deleted." is confusing; we should probably return SolsaID or UUID that was used in the request instead.&lt;/s&gt; (Dealt with in issue #274)

Also, even though REST SHOULD handle DELETE and PUT/POST/PATCH requests as idempotent, there should be a way to figure out from the response that the request was already carried earlier. For insertion or update requests, we SHOULD return actual insertion or update time-stamp if it is available, along with the actual request time-stamp. For DELETE, we could return a metadata item indicating that the record was deleted or that it was missing originally. In both cases the record is missing after the successful DELETE request. The same approach could be used also for POST/PUT/PATCH instead of the two-timestamp approach; a response metadata field could indicate if the state after request was achieved in that very request, or if it existed before. In both cases, the state of the database will be as returned in the data section of the response after the successful PUT/POST/PATCH. (RM#301, https://projects.ibt.lt/repositories/issues/301)

-- Disallow all operations other than READ on historic tables via the website.
   Entries in the historic tables should preferably be neither added nor modified via the website interface since these tables are populated explicitly by database triggers. What is more, the modifications of these entries are not recorded anywhere since the revision table itself is not versioned.

Of course, the safest option is to control the write access to these tables via database permissions, however, having an additional security level via the website interface would also be useful. The change could be as simple as to not generate the 'New', 'Edit' and 'Delete' buttons for the historic tables. (RM#296, https://projects.ibt.lt/repositories/issues/296)

-- Provide the means to specify if the table should be displayed in the table list.
   Currently, all tables (except the 'history' ones are displayed in the table list). However, this is not always the desired behaviour. For example, in the 'sample' database multiple 'epma_content' view tables (i.e. epma_Al2O3_content) do not make a lot of sense on their own and simply confuse the user (as noted by Aisha Kanzari). Table visibility could potentially be specified in the metadata database. (RM#295, https://projects.ibt.lt/repositories/issues/295)

-- Provide the means to order the table columns in the input form.
   Currently, fields in the input form are displayed in the same order they are defined in the SQL table. However, in some instances it would be very beneficial to specify the order of the fields. The metadata tables could potentially be used to store this information. (RM#294, https://projects.ibt.lt/repositories/issues/294)

-- Add script listing all available databases.
   (RM#266, https://projects.ibt.lt/repositories/issues/266)

++ Move  lib/*.pm to lib/RestfulDB/.
   All project-related libraries should go under the same namespace, in our case, RestfulDB. This is a prerequisite for the distribution both via CPAN and Debian packages. (RM#262, https://projects.ibt.lt/repositories/issues/262)

-- Properly document the process of uploading data in CSV/XLS/XLSX format.
   The usage of UUIDs during the data upload process should be properly documented. (RM#260, https://projects.ibt.lt/repositories/issues/260)

-- Record updates redirect to a table.
   On 2019-10-11 consensus was reached that record updates shouldn't redirect to a table. (RM#250, https://projects.ibt.lt/repositories/issues/250)

-- Implement the handling of 'sha1', 'sha256', 'sha512' column types.
   The metadata schema states that table columns marked as 'sha1', 'sha256', 'sha512' and 'md5' should store automatically generated hash values calculated using the appropriate algorithm. However, in revision 4870 only the 'md5' column type is properly implemented and the rest of the aforementioned types are silently ignored. The behaviour was tested with a table in which the md5 sum is calculated from an image. (RM#232, https://projects.ibt.lt/repositories/issues/232)

-- Scripts should not use restful.db_users table.
   When accessing the 'books' database at https://perly-gate.ibt.lt/~saulius/restful/website/books, the HTTP 500 status and error message is generated:

'SELECT *' failed: Table 'restful.db_users' doesn't exist at /home/saulius/public_html/restful/lib/Database.pm line 3865.

It is not clear why this table is accessed, since 'books' is a MySQL database; it uses the global 'restful' database for configuration.

In this case, the Perl layer should recognise that the 'users' table is not present, and continue in a fall-back mode. (RM#230, https://projects.ibt.lt/repositories/issues/230)

-- Hidden field without a name.
   A nameless hidden field is generated at source:trunk/lib/HTMLGenerator.pm@4544#L888. Is this intentional? (RM#219, https://projects.ibt.lt/repositories/issues/219)

-- Hidden fields are forced to table cells causing weird-looking HTML.
   The @map@ block at source:trunk/lib/HTMLGenerator.pm@4520#L1249 forces hidden fields (invisible radio buttons with "action") into single-column row of two-column table, producing weird-looking HTML. What is more, CGI.pm v4.26 seems to fail with @td( { ... }, undef )@, which happens when the second column is generated. (RM#218, https://projects.ibt.lt/repositories/issues/218)

-- Data is returned using awkward data formats if the table contains a 'content' column.
   The issue is present in /trunk at least since revision 4506.

Steps to reproduce:
1. Choose a table that contains a 'content' field, i.e. 'file' (https://solsa.crystallography.net/db-trunk/test_samples/file);
2. Directly query the 'content' field of a selected entry (i.e. https://solsa.crystallography.net/db-trunk/test_samples/file/1/content). This action triggers a normal download of a file as could be expected. In this specific example a proper docx file is returned;
3. Replace the 'content' field with any other field of the table (i.e. https://solsa.crystallography.net/db-trunk/test_samples/file/1/id). This again triggers a file download, however, the contents of the file matches those of  queried field. In this specific case, a docx file that consists only of the id value '1' is returned.

This is probably not the desired behaviour and simple plain-text could be served in case the mimetype for the field is not explicitly provided. (RM#217, https://projects.ibt.lt/repositories/issues/217)
