MarketPlace "supposed UTF-8 string is not valid UTF-8"

[gbonfiglio]

Description
After upgrading to 6.10, I'm back with the error described here #6390. When it previously occurred, it self-resolved after a few days and we dropped it there.

Adding --verbose doesn't actually get any relevant output, and nothing is written to log files in /var/log/one/ when this happens.

I'm happy to collect more debug info if anyone can give me an hint here. Thanks!

To Reproduce

root@gvm:~# onemarketapp  list
Unexpected error executing code for particular method, detected by Xmlrpc-c method registry code.  Method did not fail; rather, it did not complete at all.  263409-byte supposed UTF-8 string is not valid UTF-8.  Unrecognized UTF-8 initial byte value 0x96

Expected behavior
The command completes with no errors.

Details

• Affected Component: ONE
• Hypervisor: N/A
• Version: 6.10

Additional context

Progress Status

• Code committed
• Testing - QA
• Documentation (Release notes - resolved issues, compatibility, known issues)

[tinova]

what OS are you on? Cannot reproduce on my end

[gbonfiglio]

Debian 11

Is there any way to look a bit deeper at what's throwing this error? I'm coming empty handed across all debug / verbosity / logging options here.

[gbonfiglio]

One thing, in case it rings a bell to you - onemarketapps list throws the error above, but I'm able to see the list of IDs through onemarket show:

root@gvm:~# onemarket show 0
MARKETPLACE 0 INFORMATION
ID             : 0
NAME           : OpenNebula Public
USER           : oneadmin
GROUP          : oneadmin
STATE          : ENABLED
MARKET_MAD     : one

MARKETPLACE CAPACITY
TOTAL:         : 0M
FREE:          : 0M
USED:          : 0M

PERMISSIONS
OWNER          : uma
GROUP          : u--
OTHER          : u--

MARKETPLACE TEMPLATE
DESCRIPTION="OpenNebula Systems MarketPlace"
MARKET_MAD="one"

MARKETAPPS
435
[...]
509

(I only have the default MarketPlace and Docker's, which is empty)

[paczerny]

Hello @gbonfiglio,
we had some charset/locale issues with the DB in the past, it looks like it is carried over by the upgrade.

Identify corrupted app ID
Let's try to identify the app, with the wrong locale, in the onemarket list I can see app IDs from 435..509, try to execute:

for i in {435..509}; do onemarketapp show $i; done

Note the app ID, which fails to display, later I'll refer to this app as <corrupted_app_id>

Get more info to help us solve the issue
Now let's try to identify the issue, please connect to the MySQL DB and send as output of the following commands:

$ mysql -s -u <mysql_username> -p<mysql_password> opennebula
mysql> select * from local_db_versioning;
mysql> SELECT default_character_set_name,default_collation_name FROM information_schema.SCHEMATA WHERE schema_name = "opennebula";
mysql> SELECT CCSA.character_set_name,CCSA.collation_name FROM information_schema.`TABLES` T,information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation A
ND T.table_schema = "opennebula" AND T.table_name = "marketplaceapp_pool";
mysql> select * from marketplaceapp_pool where oid=<corrupted_app_id>;

You can find your MySQL username and password in /etc/one/oned.conf

Workaround to solve the issue
It should be enough to remove the corrupted app from the DB, the OpenNebula will reload the app in 10 minutes.

$ mysql -s -u <mysql_username> -p<mysql_password> opennebula
mysql> delete from marketplaceapp_pool where oid=<corrupted_app_id>;

Run onedb fsck to fix market reference to deleted app

[gbonfiglio]

Hey Pavel, thanks for the update here - what would I be looking for exactly in the onemarketapp show cycle you shared above? It prints a lot of stuff, but nothing in stderr. I've also grepped "error, UTF-8, xmlrpc" and nothing is returned.

[gbonfiglio]

First two outputs in the meantime:

MariaDB [opennebula]> select * from local_db_versioning;
+-----+---------+------------+-------------------------------------------------------------------------------+----------+
| oid | version | timestamp  | comment                                                                       | is_slave |
+-----+---------+------------+-------------------------------------------------------------------------------+----------+
|   0 | 5.4.1   | 1524907797 | OpenNebula 5.4.6 daemon bootstrap                                             |        0 |
|   1 | 5.6.0   | 1532182703 | Database migrated from 5.4.1 to 5.6.0 (OpenNebula 5.6.0) by onedb command.    |        0 |
|   2 | 5.8.0   | 1554844610 | Database migrated from 5.6.0 to 5.8.0 (OpenNebula 5.8.0) by onedb command.    |        0 |
|   3 | 5.10.0  | 1575286458 | Database migrated from 5.8.0 to 5.10.0 (OpenNebula 5.9.80) by onedb command.  |        0 |
|   4 | 5.12.0  | 1607810884 | Database migrated from 5.10.0 to 5.12.0 (OpenNebula 5.12.0) by onedb command. |        0 |
|   5 | 6.0.0   | 1619865534 | Database migrated from 5.12.0 to 6.0.0 (OpenNebula 6.0.0) by onedb command.   |        0 |
|   6 | 6.2.0   | 1657208836 | Database migrated from 6.0.0 to 6.2.0 (OpenNebula 6.2.0) by onedb command.    |        0 |
|   7 | 6.4.0   | 1657210916 | Database migrated from 6.2.0 to 6.4.0 (OpenNebula 6.4.0) by onedb command.    |        0 |
|   8 | 6.6.0   | 1674053725 | Database migrated from 6.4.0 to 6.6.0 (OpenNebula 6.6.0) by onedb command.    |        0 |
|   9 | 6.8.0   | 1698937850 | Database migrated from 6.6.0 to 6.8.0 (OpenNebula 6.8.0) by onedb command.    |        0 |
|  10 | 6.10.0  | 1725297252 | Database migrated from 6.8.0 to 6.10.0 (OpenNebula 6.10.0) by onedb command.  |        0 |
+-----+---------+------------+-------------------------------------------------------------------------------+----------+
11 rows in set (0.000 sec)

MariaDB [opennebula]> SELECT default_character_set_name,default_collation_name FROM information_schema.SCHEMATA WHERE schema_name = "opennebula";
+----------------------------+------------------------+
| default_character_set_name | default_collation_name |
+----------------------------+------------------------+
| latin1                     | latin1_swedish_ci      |
+----------------------------+------------------------+

[paczerny]

Hey Pavel, thanks for the update here - what would I be looking for exactly in the onemarketapp show cycle you shared above? It prints a lot of stuff, but nothing in stderr. I've also grepped "error, UTF-8, xmlrpc" and nothing is returned.

I expected the same error as onemarketapp list for one of the onemarketapp.

Are you sure the command onemarketapp list was working in 6.8 version? The upgrade to 6.10 doesn't touch the market apps, i don't see how it may fail after upgrade.

I know we had some charset issues when the MySQL changed default charset from latin1 to utf8mb4, but they should be fixed by #6113 , included in 6.8.0 version.

You can try

$ onemarket disable 0    # It will remove all apps from this marketplace
$ onemarektapp list      # To double check the error disappeard, there should be no apps
$ onemarket enable 0     # To reload all apps

If this doesn't help, please post here as a file attachment full output of select * from marketplaceapp_pool;

[gbonfiglio]

Are you sure the command onemarketapp list was working in 6.8 version?

I am not as I never tried it before, but the MarketPlace UI in Sunstone was definitely working as I pulled the Debian 12 image 10 days ago or so, now it's erroring out in the same way. This issue presented after the upgrade to 6.8, but disappeared afterwards (see #6390) with no action on my side.

---

I have now disabled both marketplaces:

root@gvm:~# onemarket list
  ID NAME                                                                                                                                                                                                              SIZE AVAIL        APPS MAD     ZONE STAT
   1 DockerHub                                                                                                                                                                                                           0M -               0 dockerh    0 off
   0 OpenNebula Public                                                                                                                                                                                                   0M -               0 one        0 off

The error persists:

root@gvm:~# onemarketapp list
Unexpected error executing code for particular method, detected by Xmlrpc-c method registry code.  Method did not fail; rather, it did not complete at all.  12175-byte supposed UTF-8 string is not valid UTF-8.  Unrecognized UTF-8 initial byte value 0x96

There are still some entries in the database, dump here: https://gist.github.com/gbonfiglio/395b785219f1be249a20e78410666b9f

(none of those entries is familiar to me, must be auto-downloaded stuff)

[gbonfiglio]

I've recovered it based on your suggestions above like this:

1 - onemarket enable 0 / onemarket enable 1
2 - systemctl stop opennebula
3 - delete from marketplaceapp_pool;
4 - onedb fsck
5 - systemctl start opennebula

Now list works. Waiting for the apps to be pulled again.

I'm curious as to how/why those entries got stuck in my DB though? They seemed to be orphaned - I still have output of onemarket show 0 and onemarket show 1 which are my only two marketplaces in my shell and neither had OIDs 102 to 408.

[paczerny]

Glad to here it works.

Thanks for the list of remaining apps - they are all from the DockerHub market. All of them registered in last month. Some of them contains UTF8 characters.

I'm afraid that if you enable DockerHub market again, the error returns.

I'll make a note for me for later, as I can't work on this now.

Possible steps to reproduce:

• backend DB with charset latin1
• Markeplace app with UTF8 characters
  • onemarketapp list may return UTF8 error
  • onemarket disable may not delete apps with UTF8 characters

[gbonfiglio]

Thankyou! Two more things for you:

• DockerHub has always been empty for me, including while things were broken (onemarket show 1 returned no apps)
• Now 12 hours later, it's still empty (whilst the ONE MarketPlace has fully populated) and things are working
• This is my current DB dump (now that things are working): https://gist.github.com/gbonfiglio/1e4407c5a7cca3d2e641f9b797ba7a5d

Should I see something in the DockerHub marketplace?

[paczerny]

I forgot that the DockerHub marketplace was removed in release 6.10, you may safely delete it by onemarket delete DockerHub

DockerHub has always been empty for me, including while things were broken (onemarket show 1 returned no apps)

The corrupted marketapps were orphaned, because the onemarket disable DockerHub removed references from the market, but removing the app failed due to corrupted UTF8 string

[gbonfiglio]

Gone! In this case, shouldn't the 6.10 migration have killed it?

[paczerny]

Closing this, created more general issue, for possible charset bugs #6709