Support

Pre-sales

#38789 Changing Database Collation Joomla 4 Admin Tools? Advice required

Posted in ‘Pre-sales and Account Questions’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Latest post by nicholas on Tuesday, 28 March 2023 06:40 CDT

[email protected]

Hi I have setup a Joomla 4 site which is using Fabrik.

The site is in a live state with multiple submissions going into the Fabrik database tables.

I am having an issue with some users when they submit text they get an error "Incorrect String Value XXXX  for column XXXXX at row 1"

The fabrik form is taking a mixture of

  • text fields
  • text area
  • file uploads (pdf, png, jpg)
  • radio fields
  • checkbox fields

The tables it seems are a mixture of collations:

  • Joomla Core tables  utfmb3_general_ci
  • Fabrik tables some are latin1_swedish_ci and some are utf8mb4_unicode_ci
  • The columns are also a mixture of these 3 collations.

Database Info

  • Server: Localhost via UNIX socket
  • Server type: MySQL
  • Server connection: SSL is not being used
  • Server version: 8.0.32 - MySQL Community Server - GPL
  • Protocol version: 10
  • Server charset: UTF-8 Unicode (utf8mb4)

Should I update the entire database tables and columns to use the same collation? I assume utf8mb4_unicode_ci would be best?

Are there any "dangers" of making this change to my system? E.g can existing records in the system become corrupted or inaccessible in a different collation?

I will of course create backups!

I appreciate any advice.

nicholas
Akeeba Staff
Manager

Joomla Core tables  utfmb3_general_ci

Yup. That's your problem. UTF8MB3 is the same as the plain old UTF-8 encoding, with up to three bytes per character. Some four-byte characters such as Emoji (e.g. 🏳️), Chinese characters etc cannot be encoded and cause a database error.

Should I update the entire database tables and columns to use the same collation? I assume utf8mb4_unicode_ci would be best?

Yes. This is the recommended encoding since Joomla 3.7. I had co-authored the code which made that possible in Joomla, with the all important finishing touches added by Richard Fath.

Are there any "dangers" of making this change to my system? E.g can existing records in the system become corrupted or inaccessible in a different collation?

No, nothing of the short. When you are going to a "wider" encoding (one which can encode more characters) you are not going to lose data. Only going to a "narrower" encoding (e.g. from four byte UTF8MB4 to  three byte UTF-8) you will lose data since the characters which cannot be encoded in the new encoding will be converted to question marks.

The other question is, of course, the PHP and Joomla version. That said, PHP version 5.4 and later do support the full Unicode charset internally, so they won't have a problem. Joomla 3.7 and later do support UTF8MB4 so, again, not a problem there.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

[email protected]

Thank you for the detailed reply Nicholas I really appreciate your time and advice.

nicholas
Akeeba Staff
Manager

You're welcome!

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!