Showing posts with label mysqlsh. Show all posts
Showing posts with label mysqlsh. Show all posts

Friday, 4 February 2022

Running BOM-encoded .sql file in MySQL shell gives "ERROR: 1064 (42000): You have an error in your SQL syntax"

I wanted to import a Northwind DB into my local instance of MySQL and found on one Google Code page a file with all necessary commands: Northwind.MySQL5.sql. I tried to execute it via MySQL Shell (mysqlsh) but got an error:

ERROR: 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '# ---------------------------------------------------------------------- #
#' at line 1


After replacing the entire content of this sql file with simple 

select User from mysql.user;

...I was still getting the same error. When running this command directly in SQL shell prompt, it was successful. But when passing the file via \source in the prompt or --file as mysqlsh argument, the error would appear. So something was wrong with the file or the way it's being passed to MySQL shell.

After some fruitless trials I created a new file and placed the same command and this time it worked fine! 



I then compared the HEX content of both files and I noticed the difference: troublemaker file was starting with Byte order mark (BOM) byte sequence: 0xefbbbf


 
I could not find any document which confirms that MySQL shell does not ignore BOM in .sql files but I found the following:


mysql ignores Unicode byte order mark (BOM) characters at the beginning of input files. Previously, it read them and sent them to the server, resulting in a syntax error. Presence of a BOM does not cause mysql to change its default character set. To do that, invoke mysql with an option such as --default-character-set=utf8.

Seems that this is not the case for mysqlsh.


       MySQL Shell’s JSON import utility importJSON() now
       handles UTF-8 encoded files that include a BOM (byte mark
       order) at the start, which is the sequence 0xEF 0xBB
       0xBF. As a workaround in earlier releases, remove this
       byte sequence, which is not needed. (Bug #30993547, Bug
       #98836)


The MySQL implementation of UCS-2, UTF-16, and UTF-32 stores characters in big-endian byte order and does not use a byte order mark (BOM) at the beginning of values. Other database systems might use little-endian byte order or a BOM. In such cases, conversion of values needs to be performed when transferring data between those systems and MySQL. The implementation of UTF-16LE is little-endian.

MySQL uses no BOM for UTF-8 values.
 
 
Also make sure (since I use PHP and this had tripped me up a couple of times so I thought I'd mention it here) all your script files are UTF8 (without BOM)
 

“Table Data Import Wizard fails on UTF-8 encoded file with BOM.”

In any case, I came to conclusion that .sql files intended to be executed by MySQL should not start with BOM character.


We can remove BOM character as here:

$ sed -i '1s/^\xef\xbb\xbf//' troublemaker.sql 
 
To check it:

$ xxd troublemaker.sql
00000000: 7365 6c65 6374 2055 7365 7220 6672 6f6d  select User from
00000010: 206d 7973 716c 2e75 7365 723b             mysql.user;


And finally, this troublemaker is not making troubles anymore:


 MySQL  172.17.0.3:3306 ssl  northwind  SQL > \source troublemaker.sql
+------------------+
| User             |
+------------------+
| root             |
| mysql.infoschema |
| mysql.session    |
| mysql.sys        |
| root             |
+------------------+
5 rows in set (0.0014 sec)


Monday, 31 January 2022

How to connect to MySQL database from Ubuntu command line




We first need to install MySQL shell.
 
I'm running my test db from within a Docker container as:
 
$ docker run \
--name my-mysql \
--rm \
-p 3306:3306 \
-e MYSQL_ROOT_PASSWORD=root \
-v mysql-volume:/var/lib/mysql \
-d mysql:latest \
--sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
 
 
To test that DB is up and running we can use telnet:
 
$ telnet 127.0.0.1 3306
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
J
8.0.27NwogIs�,
              2)IqzZ%caching_sha2_password2#08S01Got timeout reading communication packetsConnection closed by foreign host.
 
 
To connect MySQL shell running directly on my local host:
 
$ mysqlsh --mysql -u root -proot -h127.0.0.1
MySQL Shell 8.0.28

Copyright (c) 2016, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.
WARNING: Using a password on the command line interface can be insecure.
Creating a Classic session to 'root@127.0.0.1'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 64
Server version: 8.0.27 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.
 MySQL  127.0.0.1:3306 ssl  JS >
 
As we can see above, the default execution mode is set to JS (JavaScript). If we want to execute SQL commands, we need to change execution mode to SQL which can be done with \sql command (see all mysqlsh commands here):
 
MySQL  127.0.0.1:3306 ssl  JS > \sql
Switching to SQL mode... Commands end with ;
MySQL  127.0.0.1:3306 ssl  SQL >
 
Alternatively, we can pass --sql argument when running mysqlsh in order to start it in SQL mode from the very beginning (connection protocol, classic or X protocol is autodetected):
 
mysqlsh --sql ...
 
We can now execute SQL queries: 
 
MySQL  127.0.0.1:3306 ssl  SQL > show databases;
+--------------------+
| Database           |
+--------------------+
| my_custom_schema    |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.0019 sec)
 MySQL  127.0.0.1:3306 ssl  SQL >


 
 MySQL  127.0.0.1:3306 ssl  SQL > use mysql
Default schema set to `mysql`.
Fetching table and column names from `mysql` for auto-completion... Press ^C to stop.

 MySQL  127.0.0.1:3306 ssl  mysql  SQL > show tables;
+------------------------------------------------------+
| Tables_in_mysql                                      |
+------------------------------------------------------+
| columns_priv                                         |
| component                                            |
| db                                                   |
| default_roles                                        |
| engine_cost                                          |
| func                                                 |
| general_log                                          |
| global_grants                                        |
| gtid_executed                                        |
| help_category                                        |
| help_keyword                                         |
| help_relation                                        |
| help_topic                                           |
| innodb_index_stats                                   |
| innodb_table_stats                                   |
| password_history                                     |
| plugin                                               |
| procs_priv                                           |
| proxies_priv                                         |
| replication_asynchronous_connection_failover         |
| replication_asynchronous_connection_failover_managed |
| replication_group_configuration_version              |
| replication_group_member_actions                     |
| role_edges                                           |
| server_cost                                          |
| servers                                              |
| slave_master_info                                    |
| slave_relay_log_info                                 |
| slave_worker_info                                    |
| slow_log                                             |
| tables_priv                                          |
| time_zone                                            |
| time_zone_leap_second                                |
| time_zone_name                                       |
| time_zone_transition                                 |
| time_zone_transition_type                            |
| user                                                 |
+------------------------------------------------------+
37 rows in set (0.0040 sec)

 MySQL  127.0.0.1:3306 ssl  mysql  SQL > describe user;
+--------------------------+-----------------------------------+------+-----+-----------------------+-------+
| Field                    | Type                              | Null | Key | Default               | Extra |
+--------------------------+-----------------------------------+------+-----+-----------------------+-------+
| Host                     | char(255)                         | NO   | PRI |                       |       |
| User                     | char(32)                          | NO   | PRI |                       |       |
| Select_priv              | enum('N','Y')                     | NO   |     | N                     |       |
| Insert_priv              | enum('N','Y')                     | NO   |     | N                     |       |
| Update_priv              | enum('N','Y')                     | NO   |     | N                     |       |
| Delete_priv              | enum('N','Y')                     | NO   |     | N                     |       |
| Create_priv              | enum('N','Y')                     | NO   |     | N                     |       |
| Drop_priv                | enum('N','Y')                     | NO   |     | N                     |       |
| Reload_priv              | enum('N','Y')                     | NO   |     | N                     |       |
| Shutdown_priv            | enum('N','Y')                     | NO   |     | N                     |       |
| Process_priv             | enum('N','Y')                     | NO   |     | N                     |       |
| File_priv                | enum('N','Y')                     | NO   |     | N                     |       |
| Grant_priv               | enum('N','Y')                     | NO   |     | N                     |       |
| References_priv          | enum('N','Y')                     | NO   |     | N                     |       |
| Index_priv               | enum('N','Y')                     | NO   |     | N                     |       |
| Alter_priv               | enum('N','Y')                     | NO   |     | N                     |       |
| Show_db_priv             | enum('N','Y')                     | NO   |     | N                     |       |
| Super_priv               | enum('N','Y')                     | NO   |     | N                     |       |
| Create_tmp_table_priv    | enum('N','Y')                     | NO   |     | N                     |       |
| Lock_tables_priv         | enum('N','Y')                     | NO   |     | N                     |       |
| Execute_priv             | enum('N','Y')                     | NO   |     | N                     |       |
| Repl_slave_priv          | enum('N','Y')                     | NO   |     | N                     |       |
| Repl_client_priv         | enum('N','Y')                     | NO   |     | N                     |       |
| Create_view_priv         | enum('N','Y')                     | NO   |     | N                     |       |
| Show_view_priv           | enum('N','Y')                     | NO   |     | N                     |       |
| Create_routine_priv      | enum('N','Y')                     | NO   |     | N                     |       |
| Alter_routine_priv       | enum('N','Y')                     | NO   |     | N                     |       |
| Create_user_priv         | enum('N','Y')                     | NO   |     | N                     |       |
| Event_priv               | enum('N','Y')                     | NO   |     | N                     |       |
| Trigger_priv             | enum('N','Y')                     | NO   |     | N                     |       |
| Create_tablespace_priv   | enum('N','Y')                     | NO   |     | N                     |       |
| ssl_type                 | enum('','ANY','X509','SPECIFIED') | NO   |     |                       |       |
| ssl_cipher               | blob                              | NO   |     | NULL                  |       |
| x509_issuer              | blob                              | NO   |     | NULL                  |       |
| x509_subject             | blob                              | NO   |     | NULL                  |       |
| max_questions            | int unsigned                      | NO   |     | 0                     |       |
| max_updates              | int unsigned                      | NO   |     | 0                     |       |
| max_connections          | int unsigned                      | NO   |     | 0                     |       |
| max_user_connections     | int unsigned                      | NO   |     | 0                     |       |
| plugin                   | char(64)                          | NO   |     | caching_sha2_password |       |
| authentication_string    | text                              | YES  |     | NULL                  |       |
| password_expired         | enum('N','Y')                     | NO   |     | N                     |       |
| password_last_changed    | timestamp                         | YES  |     | NULL                  |       |
| password_lifetime        | smallint unsigned                 | YES  |     | NULL                  |       |
| account_locked           | enum('N','Y')                     | NO   |     | N                     |       |
| Create_role_priv         | enum('N','Y')                     | NO   |     | N                     |       |
| Drop_role_priv           | enum('N','Y')                     | NO   |     | N                     |       |
| Password_reuse_history   | smallint unsigned                 | YES  |     | NULL                  |       |
| Password_reuse_time      | smallint unsigned                 | YES  |     | NULL                  |       |
| Password_require_current | enum('N','Y')                     | YES  |     | NULL                  |       |
| User_attributes          | json                              | YES  |     | NULL                  |       |
+--------------------------+-----------------------------------+------+-----+-----------------------+-------+
51 rows in set (0.0046 sec)

 MySQL  127.0.0.1:3306 ssl  mysql  SQL > select User, Host from user;
+------------------+-----------+
| User             | Host      |
+------------------+-----------+
| root             | %         |
| mysql.infoschema | localhost |
| mysql.session    | localhost |
| mysql.sys        | localhost |
| root             | localhost |
+------------------+-----------+
5 rows in set (0.0035 sec)


If we know in advance the schema (database) that we want to use, we can pass its name to --schema or --database argument. Also, if we want to save the output of all commands from the interactive session into a file, we can specify --pager command (e.g. tee):

$ mysqlsh --host=127.0.0.1 --user=root --password=root --sql --schema=sys --pager="tee mysqlsh.out"
MySQL Shell 8.0.28

Copyright (c) 2016, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.
WARNING: Using a password on the command line interface can be insecure.
Creating a session to 'root@127.0.0.1/sys'
Fetching schema names for autocompletion... Press ^C to stop.
Fetching table and column names from `sys` for auto-completion... Press ^C to stop.
Your MySQL connection id is 66
Server version: 8.0.27 MySQL Community Server - GPL
Default schema set to `sys`.
 MySQL  127.0.0.1:3306 ssl  sys  SQL > show tables;
+-----------------------------------------------+
| Tables_in_sys                                 |
+-----------------------------------------------+
| host_summary                                  |
| host_summary_by_file_io                       |
...
| x$waits_by_user_by_latency                    |
| x$waits_global_by_latency                     |
+-----------------------------------------------+
101 rows in set (0.0060 sec)
 MySQL  127.0.0.1:3306 ssl  sys  SQL > \quit
Bye!

Let's check the output file now:


$ cat mysqlsh.out
+-----------------------------------------------+
| Tables_in_sys                                 |
+-----------------------------------------------+
| host_summary                                  |
| host_summary_by_file_io                       |
...
| x$waits_by_user_by_latency                    |
| x$waits_global_by_latency                     |
+-----------------------------------------------+
101 rows in set (0.0060 sec)


 

To execute multiple commands in a single session and store their outputs in order of execution into a (text) file we need first to write a .sql file which lists all commands.

mysql-queries.sql:

select now() as '';
select 'This is a mysqlsh session output' as'';
show tables;

Then we can set it as a source file and redirect output into a file:

$ mysqlsh --host=127.0.0.1 --user=root --password=root --sql --schema=sys --file=mysql-queries.sql > mysqlsh.out
WARNING: Using a password on the command line interface can be insecure.

Let's check the output file:

$ cat mysqlsh.out

2022-01-31 16:36:49

This is a mysqlsh session output
Tables_in_sys
host_summary
host_summary_by_file_io
host_summary_by_file_io_type
...
x$waits_by_user_by_latency
x$waits_global_by_latency

Another way to execute multiple commands and pipe output into a file is by listing SQL commands one by another, separated by semicolon.
 
# mysqlsh \
--host=$MYSQL_DB_HOST \
--port=$MYSQL_DB_PORT \
--user=root \
--password=root \
--sql \
--execute='use mysql; select User from user;' > ./out/$SQL_FILE.out

The queries above are equal to:

select User from mysql.user;


MySQL DB ports and connection protocols

 
It is possible to check on which port DB listens to:
 
$ docker container inspect mysql-demo | jq .[].NetworkSettings
{
  "Bridge": "",
  "SandboxID": "ad84f18f89bb51dd6c694357346633aae3350e7333f5eadd4296aa8a81fba96a",
  "HairpinMode": false,
  "LinkLocalIPv6Address": "",
  "LinkLocalIPv6PrefixLen": 0,
  "Ports": {
    "3306/tcp": [
      {
        "HostIp": "0.0.0.0",
        "HostPort": "3307"
      }
    ],
    "33060/tcp": null
  },
  "SandboxKey": "/var/run/docker/netns/ad84f18f89bb",
  "SecondaryIPAddresses": null,
  "SecondaryIPv6Addresses": null,
  "EndpointID": "246dcd5e7ab59333868ebdeda32f8580309a9b5daeac5648f369f53a62c5c461",
  "Gateway": "172.17.0.1",
  "GlobalIPv6Address": "",
  "GlobalIPv6PrefixLen": 0,
  "IPAddress": "172.17.0.3",
  "IPPrefixLen": 16,
  "IPv6Gateway": "",
  "MacAddress": "02:42:ac:11:00:03",
  "Networks": {
    "bridge": {
      "IPAMConfig": null,
      "Links": null,
      "Aliases": null,
      "NetworkID": "7a20c87648fe59ba77848c398b3c69e0073a294f018c9c563242302cddcba16c",
      "EndpointID": "246dcd5e7ab59333868ebdeda32f8580309a9b5daeac5648f369f53a62c5c461",
      "Gateway": "172.17.0.1",
      "IPAddress": "172.17.0.3",
      "IPPrefixLen": 16,
      "IPv6Gateway": "",
      "GlobalIPv6Address": "",
      "GlobalIPv6PrefixLen": 0,
      "MacAddress": "02:42:ac:11:00:03",
      "DriverOpts": null
    }
  }
}


Default classic MySQL protocol port is usually 3306
Default X Protocol port is usually 33060

From the output above, we can see that our DB server supports only classic MySQL protocol.


Exporting Tables

How to export tables from MySQL Database | My Public Notepad


Importing Tables