Support

Akeeba Ticket System

#36017 Fetch Email Plug In - attachment size bigger than 2MB

Posted in ‘Akeeba Ticket System for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Akeeba Ticket System version
n/a

Latest post by schrammelmann on Friday, 22 October 2021 10:06 CDT

schrammelmann

Hi!

When importing e-mails, it seems that attachments are limited to 2MB.

As a result, image files larger than 2MB become unusable, as only part of the image file is saved in the ticket.

I have set the upload size in the Joomla Media Manager to 10 MB.

Maybe this is a problem in connection with the Horde Client. However, I could not identify the problem in the source code.

Many thanks in advance for your effort!

Michael

nicholas
Akeeba Staff
Manager

No, this is a problem with your PHP configuration. You can set whatever upload size in media manager. If it's larger than PHP's max_upload_size or post_max_size it will be reduced to the smaller of these two PHP configuration values.

The default settings of max_upload_size and post_max_size are, indeed, 2MB.

Also remember that Cron Jobs are handled by a different PHP executable than web requests. Each PHP executable has its own configuration file, independent from other PHP versions' or executables' configuration files. I understand that you are using a CRON job to fetch the emails which means that you need to edit the configuration for that PHP CLI version and set max_upload_size and post_max_size to 10M.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

schrammelmann

Thank you very much, you are right. I had not thought of that. I use a cron job and checked the settings with phpinfo. It was indeed the default values of 2 MB.
My hoster runs PHP within the FastCGI Process Manager (FPM) and I have to use a .user.ini File for adapting PHP settings. But I had problems including such a file. The settings were not used. Therefore I tried this command in my CRON script:
env -i /usr/bin/php -d upload_max_filesize=50M -d post_max_size=0 $cronfile --command=mailfetch --verbose >> $logfile
I checked that the parameters are applied by additionally using the parameter -i.
But, strangely, the attachments are still cut off at 2 MB.
Are you sure that this works in your environment?

nicholas
Akeeba Staff
Manager

The .user.ini file only applies to PHP-FPM. PHP-FPM uses the PHP CGI binary (it's the same binary for CGI and FastCGI) which has one set of configuration, the one that you can edit through your hosting control panel.

The CRON job runs though the PHP CLI binary. Typically, its configuration is not accessible through the hosting control panel. Moreover, the .user.ini file does not apply to it. You can either ask your host to change its configuration or, as you did, change its configuration in the command line if it is allowed.

Also, we are not entirely sure if /usr/bin/php is PHP-CLI or PHP-CGI on your server, or which PHP version it actually is. It is very possible (almost certain!) that your server runs multiple versions of PHP. While your site has been configured to use PHP 7.4 — via a .htaccess line which selects the correct PHP-FPM instance using an AddHandler or SetHandler directive — the /usr/bin/php would be the default PHP version of your server which could be a different one, e.g. PHP 7.3.

Please do ask your host to confirm the path of the PHP CLI binary for PHP 7.4 and use that in your CRON command line. Then please do ask them to change the two configuration settings I told you about in the PHP CLI configuration.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

schrammelmann

Once again you have hit the bull's eye. And thank you for the explanations.

I have contacted the support of my hoster.

On the shell/cron level, only PHP CLI is used.
With /usr/bin/php74 I can use PHP version 7.4.23.
I can only make settings via the -d parameter or via an .ini file passed with the -c parameter.

I was advised not to set post_max_size to 0 but explicitly to a value.
I should also test whether the setting memory_limit=1G solves the problem (default is 128M).

My command line call was otherwise confirmed as correct.

This resulted in the following new call:

env -i /usr/bin/php74 -d upload_max_filesize=50M -d post_max_size=50M -d memory_limit=1G $cronfile --command=mailfetch --verbose >> $logfile

Unfortunately, but with the same result as before. The attachments are limited to 2MB.

Could you test on your test environment whether files larger than 2 MB can actually be imported via Fetch Mail Plugin in your system?

I had read somewhere about problems with a 2MB limit in the Horde client. Maybe this also affects the Horde Imap client, which I think is used in the plugin.

nicholas
Akeeba Staff
Manager

I am currently working on rewriting Akeeba Ticket System for Joomla 4. The past two days I have actually been working on the email receiving feature. I have found out a few things:

  • UTF-8 subjects: this is caused partially because of how Horde decodes UTF-8 and partly legacy code from the Joomla 1.x/2.5 days. Some characters like ü can be expressed in UTF-8 in different ways. For example, ü can either be a single glyph (U with dieresis) or a combination of the U glyph and the dieresis glyph. The former works. The latter introduces an issue when creating the ticket URL alias. Now that I know what the problem is I can solve it. This will be fixed in ATS 4 for Joomla 3, it's literally adding one character in one line of code and removing another line of code.
  • Horde says it's compatible with PHP 5.3 and later. What it does NOT say is that it doesn't work correctly with PHP 7.4 and not at all with PHP 8. The PHP 7.4 compatibility issues result in the message envelope not being read correctly. On PHP 8 it bombs out completely because Horde uses a while(...each(...)) construct instead of a foreach(...). That's the kind of code people wrote in the early PHP 4.0 days. Luckily there's Rector which can address these issues. I am not 100% confident I can backport this to ATS4 for Joomla 3 but I will try...
  • Horde could not be updated anymore because it's distributed via PEAR and the Composer to PEAR bridge was removed in Composer 2.0. I need to implement a special trick to get it updated. Sure, yes, I can use an older Composer version — however, the PEAR bridge in the latest Horde adds hardcoded paths specific to my computer which means I need to do some additional workarounds to replace them with relative paths.

I hate having to use Horde but I have no choice :( Gmail / GSuite moved away from standard IMAP authentication and into XOAUTH2. The only PHP library which supports XOAUTH2 authentication is Horde. Yahoo and Microsoft now also use XOAUTH2 so it's not like I can pretend I can go back to using PHP's imap extension. Even more so when the PHP developers have said that the C library it uses internally hasn't been updated since 2007 and there's fat chance anyone will ever implement any additional features, including XOAUTH2, in it. Yikes. We are left with two options: use Horde or remove any feature that reads email. The latter is not a realistic option so Horde, with all its drawbacks, it is then!

Regarding the attachment size, I will investigate further today. I have so far found that the size of the attachment doesn't matter; Horde reads it fine. The most likely problem is the extension of the file. If it doesn't match Joomla media manager's acceptable list of extensions it won't be included in the ticket. I will take a wild guess and say that you are trying to upload a ZIP file but the zip extension is not allowed by default in Joomla; you'd have to add it manually in the Media Manager's Options page. I will also guess that it's very likely that the extension is .ZIP (uppercase) instead of .zip (lowercase) but since Joomla's file extension checks are case–sensitive it blocks the upload.

I am working on workaround for the checks on filenames and the file extensions BUT this requires an overhaul of the attachments upload feature which means that it will only be available in Akeeba Ticket System 5 for Joomla 4 only. We are no longer adding any new features to ATS4 or our Joomla 3 software in general as we explained in mid–August 2021. Joomla 3 has an expiration date of August 17th, 2023 and our clients are already moving en masse to Joomla 4. Based on our usage stats, the month–over–month use of Joomla 4 is more than doubling. In fact, if we break it down to weeks it shows an accelerating trend. There's no business sense in spending time to complete a big refactoring for ATS 4's attachments when less than half of our clients will be using Joomla 3 by the time we are done. It makes more sense to implement the new features in ATS 5 for Joomla 4.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

schrammelmann

Ohh Nicholas, I really feel bad about wasting your time.
I understand the problem and can sympathise with it.

Nevertheless, let's take a quick look at the size problem:

For me, it's all about image files. Usually jpg, png. No ZIP files. I had already added the missing file extensions in Media Manger.

The effect is that all files (except those with ü,ä etc.) are imported into the ticket. But the files are mercilessly cut off at 2 MB.

This means that I can download and view the attachments, but only half of an image with, for example, a size of 4 MB is visible. The lower part cannot be displayed because it is missing. The file is actually destroyed, but because of the image format, the upper part can still be displayed.

Actually, I have now found workarounds for all issues.
In addition, I am misusing your ticket system for a purpose for which it was not intended.

For me and my team it is a bit more complicated than before, but we can still work with it and the akeeba ticket system is much better than our old one. And besides, it is perfectly supported. :-)

So I leave it up to you whether you really want to invest any more time in the issues I reported. For most of your users it doesn't seem to be a problem.

If the cause of the 2 MB problem is on my end and I happen to find it, I will definitely report it here.

So we can close the ticket with this status if you agree.

Best regards,
Michael

 

nicholas
Akeeba Staff
Manager

Not a waste of time at all! In fact, your tickets came at the perfect time :) Email parsing is very complicated. I need to spend a day or so to “load” all that code in my mind and set up a bunch of mail servers and email accounts to troubleshoot deeper issues. As it happens, yesterday was the second day I was working exclusively on that code so what better time to troubleshoot it than now? That's why I was able to figure out the UTF-8 issue immediately. That code felt “icky”. You gave me enough information to reproduce your issue, track it down to the icky code and fix it.

Regarding the 2MB limit, I cannot reproduce it with the updated version of Horde. I actually did try it with a 2.9MB JPEG and it works great.

In the latest version of Horde the 2MB limit applies to parsing MIME–encoded email parts on the web server. However, as it has been the case since we implemented replies by email, we are asking the mail server to decode the MIME part. I confirmed that we are receiving the full binary contents of the attachment, we save it properly and that they are correct down to the last byte.

I am not sure about the older Horde version used in ATS4, though. My plan is to use the same Horde version in both versions of ATS for my sanity's sake. So... this problem is MOST LIKELY going to be resolved in the next release.

You don't need to reply to this ticket. You can either let it auto–close in 4 weeks or you can come back here and close it yourself.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

schrammelmann

I would like to choose the way of closing the ticket myself, as I would like to say a big thank you once again.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!