1

I am running a LAMP setup w/ the 64bit Raspbian OS on a Raspberry Pi4.

I am having a weird issue with 'utf8' and executing python scripts from PHP that are using a connector to my database in mariadb.

I have no issues running the scripts and modifying the database through my IDE.

I have no issues inserting a record into the same table manually with PHP:

// Create connection
$conn = new mysqli($servername, $username, $password, $database);
$sql = "INSERT INTO commandRequestLog VALUES (0, 1001, 'phpinsert', CURRENT_TIME(), CURRENT_TIME(), 0, 'testing php sql connection', null)";

I have no issues running/returning other logical python scripts in PHP with exec() that do not connect to the database using the same syntax:

exec('python3 command_processor.py '.$UserType.' '.$UserId.' '.$PyCommandLvl.' "'.$CommandHdr.'" "'.$strCommand.'"', $output);

foreach ($output as $line) {
 echo $line ."<br>";
}

That is until I introduce a connection to the database within my script:

import mysql.connector

def newConnection():
return mysql.connector.connect(host='localhost',
                               port=3306,
                               ##charset='utf8',
                               database='xxxxxxx',
                               user='xxx',
                               password='xxxxxxxx')

I am attempting to run this python script from a PHP exec():

if insert_commandRequest(intUserId, strRequestedCommand, strDescription) == -1:
    strOutput = 'sql still not working wtf...'+intUserId+strRequestedCommand+strDescription
    print(strOutput)
    
    try:
        newConn = newConnection()##Fails right here...
   except Error as error:
        return error
    finally:
        print('finally')

Then it will return this Exception Error along with the remainder of the expected output:

Character set 'utf8' unsupported ##Primary Error...
Command Requested : [asdf]
Command Function : [asdf]

I recall having an issue with utf8 initially even in my IDE, but the quick fix was to downgrade my mysql-connector-python version to 8.0.29. That was working fine everywhere up until introducing the first connection within any PHP exec().

I have gone down damn near every rabbit hole such as:

  • Trying various PHP exec functions like shell_exec()

  • Re-installing (python, python3, python modules, pip, pip3, mysql-connector-python, php)

  • Modifying my.conf mysql file to include 'utf8mb4' (and various others) for all defaults then restoring my database like 5 times. (Got a great database creation script now though :) )

  • Trial and error on various utf8 syntax

  • Using mariadb.connect instead of mysql.connector.connect which then results in all of my PHP exec()'s not returning any output

MariaDB Server version

obi@raspberrypi:~ $ sudo mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 64
Server version: 10.5.15-MariaDB-0+deb11u1 Debian 11

Current database db.opt file

default-character-set=utf8mb4
default-collation=utf8mb4_unicode_ci

Current /etc/mysql/my.cnf file:

[client-server]
# Port or socket location where to connect
# port = 3306
 socket = /run/mysqld/mysqld.sock

# Import all .cnf files from configuration directory
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4


[mysqld]
collation-server = utf8mb4_unicode_ci
init-connect='SET NAMES utf8mb4'
character-set-server = utf8mb4

Python3 and PHP package info:

obi@raspberrypi:~ $ sudo apt-cache show python3
Package: python3
Source: python3-defaults
Version: 3.9.2-3
Installed-Size: 89
Maintainer: Matthias Klose <[email protected]>
Architecture: arm64
Replaces: python3-minimal (<< 3.1.2-2)
Provides: python3-profiler
Depends: python3.9 (>= 3.9.2-0~), libpython3-stdlib (= 3.9.2-3)
Pre-Depends: python3-minimal (= 3.9.2-3)
Suggests: python3-doc (>= 3.9.2-3), python3-tk (>= 3.9.2-0~), python3-venv (>= 3.9.2-3)
Description-en: interactive high-level object-oriented language (default python3 version)
 Python, the high-level, interactive object oriented language,
 includes an extensive class library with lots of goodies for
 network programming, system administration, sounds and graphics.
 .
 This package is a dependency package, which depends on Debian's default
 Python 3 version (currently v3.9).
Description-md5: 691450e63b8789f4ba89699b19c60642
Multi-Arch: allowed
Homepage: https://www.python.org/
Cnf-Extra-Commands: python
Cnf-Priority-Bonus: 5
Tag: devel::interpreter, devel::lang:python, devel::library,
 implemented-in::c, implemented-in::python, role::devel-lib,
 role::program, role::shared-lib
Section: python
Priority: optional
Filename: pool/main/p/python3-defaults/python3_3.9.2-3_arm64.deb
Size: 37880
MD5sum: 3cb918cd19c9186711e9c0395e668277
SHA256: 79197285d25e73a2a07667efe80af152dd932ac5ef3e13717f1ac824d111ea81

obi@raspberrypi:~ $ sudo apt-cache show php
Package: php
Source: php-defaults (76)
Version: 2:7.4+76
Installed-Size: 13
Maintainer: Debian PHP Maintainers <[email protected]>
Architecture: all
Depends: php7.4
Description-en: server-side, HTML-embedded scripting language (default)
 PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used
 open source general-purpose scripting language that is especially suited
 for web development and can be embedded into HTML.
 .
 This package is a dependency package, which depends on latest stable
 PHP version (currently 7.4).
Description-md5: 99c58fa41ae5c5908fd17241a53bfdd9
Section: php
Priority: optional
Filename: pool/main/p/php-defaults/php_7.4+76_all.deb
Size: 6340
MD5sum: 33859926f35bdfdc784d9e7312d16a92
SHA256: 10ca22712771dc343fb6600cbdbc88069069467fcc5c85782ee89abbc8c81f59

I know the simple solution would be to just run all of my database connections through PHP instead of Python. But I already wrote everything in Python and everything else is working just fine, I feel like I should be able to get over this hurdle, but I am pretty stuck.

Any tips are much appreciated, cheers!

4
  • Try utf8mb4 instead of utf8 in your python script? Commented Oct 20, 2022 at 19:55
  • @aynber I have tried that as well. Just did it again to be sure and received this: [1273 (HY000): Unknown collation: 'utf8mb4_0900_ai_ci']. I have tried a grep in /etc/mysql and /var/lib/mysql, and unless I am doing it wrong, I can't seem to find that string specified anywhere.... Commented Oct 20, 2022 at 20:17
  • I'm pretty sure you didn't mean that you're using Python 8 :-) But seriously, this does feel like maybe a Python version mismatch. Can you verify that the version of Python you're calling from your PHP script is the same version you downgraded to? It's not at all unusual to have 2 or 3 or even 4 versions of the Python binary floating around on the same system, so I would be sure to be very thorough in checking to make sure your PHP script is accessing the right one. Commented Oct 21, 2022 at 7:31
  • @MayorofthePlattenbaus yah sorry I meant mysql-connector-python is on the version 8.0.29. Fixed that thanks. I will have to check that out, thanks for the tip. For now I have written a PHP SQL function until I resolve python connector. Commented Oct 22, 2022 at 0:56

1 Answer 1

1

This is a known issue introduced by MySQL Connector/Python 8.0.30 with MariaDB when using the pure Python implementation.

In your case you have the scenario of not having the C extension implementation of Connector/Python, since you're using Raspbian OS.

You have two choices, one is to downgrade to version 8.0.29 other is compile MySQL Connector/Python from source in order to build the C extension, see https://dev.mysql.com/doc/connector-python/en/connector-python-installation-source.html

This issue will be addressed in the upcoming release.

Sign up to request clarification or add additional context in comments.

4 Comments

That would be great advice if I was not already running version 8.0.29.... obi@raspberrypi:~ $ pip show mysql-connector-python Name: mysql-connector-python Version: 8.0.29
I will try the download from source solution as well, thanks for the info.
Thank you so much for this answer. I had been unable to connect to one of my databases after reinstalling my OS and all Python packages and I'd gotten frustrated trying to figure out where utf8mb4_0900_ai_ci was coming from.
This issue was fixed in 8.0.32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.