Akeeba Ticket System

Monday, 31 July 2023 14:14 CDT

Hello again,

After all of the past issues getting ATS categories and mailfetch issues worked out we are ready to finally migrate data from our J3/ATS4 site to our J4/ATS5 site. Previously, I had tested the procedures you provided in a past ticket a few months ago successfully.

However, I have now run into an issue using ATS 5.2.8.

Can you confirm that uninstalling ATS 5.2.8 should still leave any previously created created categories, custom fields, and menu items? You referred to this as the "naive" way the uninstallation works.

In my first test using ATS 5.2.8 I found that the previously created categories and custom fields are no longer present. Is this a change in the uninstall behavior? If so this changes the data migration strategy and we would need alternative methods.

Here were my high level steps copied from your recommendation in previous ticket:

(Legacy Server) Backup current production ATS DB tables
(New Server) Install and configure ATS 5.2.8 including all categories, custom fields and menu items. It should be noted that ATS 5.2.8 was not a fresh install but rather a culmination of various updates, and dev builds to resolve all of the previous bugs we have hit.
(New Server) Current state full backup just in case
(New Server) Uninstall ATS 5.2.8
(New Server) Install ATS 4.0.8 special build provided by you to bypass the server requirement check to allow install on J4.2.x and higher
(New Server) Restore all #_ats_tables from backup archive created in Step 1
(New Server) Install ATS 5.2.8

It seems that after Step 7 the previously created categories and custom fields in step 2 are not present.

I DO see that there are tickets from the restore process in the dashboard summary. However, since there are no Ticket categories they are not appearing if I select Tickets in the admin component. I have attached some screenshots.

Any suggestions would be greatly appreciated.

Thanks,

Eric

Tuesday, 01 August 2023 07:35 CDT

[email protected]

Hello,

I believe that this issue may be related to the max_input_vars setting in php.ini.

During the restore process the php.ini file was overwritten and the max_input_vars was reset to the default of 1000.

This prevented ATS 5.x from being able to create any categories, manually or during the upgrade from ATS4 to ATS5.

I am still in the process of testing the end to end but wanted to prevent any unnecessary investigations on your part until I have conclusive results.

Thanks for your support and patience!

Eric

Tuesday, 01 August 2023 08:47 CDT

[email protected]

Hello,

I have tested the following procedure multiple times after ensuring that the php.ini setting max_input_vars = 5000 . In addition I have verified that I can add new or modify existing ATS 5.2.8 categories prior to starting the data migration process from J3/ATS4.

I can confirm that uninstalling ATS 5.2.8 is removing the com_ats categories. See this screen recording:

https://watch.screencastify.com/v/TvuapyGeDFYMkMzyqdJ1

Is this by design with the latest versions of ATS?

Do you have any other suggested workarounds to get data from J3/ATS4 migrated to J4/ATS5?

Thanks,

Eric

Tuesday, 01 August 2023 08:48 CDT

nicholas

The uninstallation in ATS 5 should remove secondary Joomla! artefacts, including custom fields and their groups and values, UCM content types and entries (used by tags), categories, and mail templates.

You can, however, change that by editing the file administrator/components/com_ats/sql/uninstall.mysql.utf8.sql. Each block of SQL code has a comment describing what it does. You should remove everything from -- Remove ticket custom field values onwards, except the two blocks about UCM entries (because the installation database code tries to add these entries). You will only lose tags, but you get to keep categories, field groups, groups, and mail templates. I believe that's what you want.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Tuesday, 01 August 2023 08:54 CDT

[email protected]

Nicholas,

Thank you for the explanation. Yes, I would like to keep categories, mail templates, fields and field groups. I am assuming that this is a change in behavior from earlier versions of ATS5?

The previous guidance you provided regarding migrating from J3/ATS4 to J4/ATS5 was leveraging the fact that the uninstall scripts did NOT remove categories, mail templates, fields and field groups.

I will give it a try.

Thanks!

Eric

Tuesday, 01 August 2023 09:34 CDT

nicholas

I am assuming that this is a change in behavior from earlier versions of ATS5?

Yes. The “naïve” way the uninstallation worked meant that you ended up with invisible leftovers you wouldn't know how to remove. The Joomla! Extensions Directory (JED) terms of service require listed extensions to perform a complete uninstallation, so we had to change the behavior to be able to finally list ATS on the JED.

The previous guidance you provided regarding migrating from J3/ATS4 to J4/ATS5 was leveraging the fact that the uninstall scripts did NOT remove categories, mail templates, fields and field groups.

Yep. Back then (was it, like, a year ago?) the uninstallation was very naïve.

All of our Joomla 4 products were developed in “waves”. First, we got the minimum viable compatibility with Joomla 4 in our Joomla 3 software, knowing this will be necessary for upgrades. This started about 3 years before Joomla 4 was released (when the second alpha was out) and continued until 9 months prior to Joomla 4.0 (with a few adjustments a couple weeks prior to the release).

Then, we created native Joomla! 4 versions of everything, from scratch, between 9 months before and 9 months after the release of Joomla! 4.0 — ATS was, in fact, one of the last extensions to be converted due to the insane scope of features that had to be re–implemented, paying the technical debt from Joomla! 2.5 onwards (with interest, I might add).

Then we focused on improvements, changes in under–the–hood API features introduced in Joomla 4.1 / 4.2 / 4.3. That is when I refined the uninstallation to meet the JED standards so ATS could be finally listed there, a full decade after we first released it as a standalone extension.

The most recent wave is the preparation for Joomla! 4.4 and 5.0. And yes, our software is already compatible with Joomla! 5.0.

Somewhere in there we overlapped another “wave” of PHP 8.2 compatibility updates, and PHP 8.3 compatibility updates. Yes, PHP 8.3 is only in beta as of last week but we already support it.

The fact that clients don't notice any of that most of the time is deliberate. We do everything we can to ensure that any necessary migrations are automatic. We want you to focus on building and managing your sites, not fighting with our software. In a very few, quite rare, occasions the seamless façade has to give way because the change is absolutely necessary and impossible to hide in any reasonable way. The last half decade there have only been three: the migration from Akeeba Backup 8 to 9 (which is still a couple of mouse clicks), the upgrade from ATS 4 to 5 (which is unfortunately and inevitably painful — we had to go through the same with our own site), and the change in ATS' uninstallation in version 5.1 and later. In both cases, we had to pay the technical debt accrued over 10 years. If you're wondering, nope, the other extensions had not accrued that much technical debt because they are ultimately much simpler and we could do heavy refactoring without anyone being the wiser.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Tuesday, 01 August 2023 09:57 CDT

[email protected]

Nicholas,

I have tried multiple times and I have found that the ATS categories are still being removed even though the uninstall script was edited as shown below. I did verify that the ATS mail templates, fields and field groups are still present however.

Is there possibly another uninstall script in which the com_ats categories are removed?

https://watch.screencastify.com/v/vKT7JKlVMF0U1Y2u4wFu

Thanks in advance for any suggestions!

Eric

/**
* @package ats
* @copyright Copyright (c)2011-2023 Nicholas K. Dionysopoulos / Akeeba Ltd
* @license GNU General Public License version 3, or later
*/

-- Remove our tables
DROP TABLE IF EXISTS `#__ats_tickets`;
DROP TABLE IF EXISTS `#__ats_posts`;
DROP TABLE IF EXISTS `#__ats_attachments`;
DROP TABLE IF EXISTS `#__ats_managernotes`;
DROP TABLE IF EXISTS `#__ats_cannedreplies`;
DROP TABLE IF EXISTS `#__ats_autoreplies`;

/**
* @package ats
* @copyright Copyright (c)2011-2023 Nicholas K. Dionysopoulos / Akeeba Ltd
* @license GNU General Public License version 3, or later
*/

-- Remove the UCM content type for tickets
DELETE FROM `#__content_types` WHERE `type_alias` = 'com_ats.ticket';

-- Remove the UCM content entries for tickets (used by tags).
-- Note that we cannot remove the tags themselves; tags are global, not per content type!
DELETE FROM `#__ucm_content` WHERE `core_type_alias` = 'com_ats.ticket';

Tuesday, 01 August 2023 10:09 CDT

[email protected]

Nicholas,

Sorry our replies crossed as I didn't refresh my browser before replying.

My hope is that you don't take any of my tickets as complaining about your product. We couldn't function without Akeeba. Backup makes my life so much easier.

I appreciate the complexity of your products and can't imagine the complexity of trying to manage things that outside of your direct control. Joomla core, other third party extensions etc. In addition you have customers with varying use cases and some (like me) trying to use Joomla for an entire IT stack.

Any help you can provide would be appreciated and if we are getting into outside of the normal support agreement, I would welcome a discussion regarding custom services to get our data migrated. The good news is that we are planning on queueing up tickets on the mailbox during the cutover. So we can do the restore operation 1 time during an "outage window".

Much appreciated!

Eric

Tuesday, 01 August 2023 12:27 CDT

nicholas

It's actually been very interesting providing support for you guys as we got to see some issues cropping up with a larger scale deployment of ATS outside of the typical use cases we had so far. At no point did I get the impression that you're criticising the product.

In fact, you are wonderfully kind, very precise in what you want to do, what you've done to self-resolve the issues, and what happened next. This makes it very easy for us to understand what the use case is, what is going on, and how to help you — even if it means implementing an entirely new class of features (like the very detailed permissions). You are also very understanding about the things we say we cannot do and why we cannot do them. This is the ideal support interaction for us. If only that was the norm! I trust that you know exactly how things get in tech support :)

I have no reason to charge you extra. Yes, you may make a bit more work for us, but you help us improve our software, and you're a joy to work with on top of that. I am happy with that!

By the way, the reason I explained the development process was to provide some more insight, since this is a public ticket.

For your migration, I strongly recommend investing in automating it and testing it thoroughly on a test server, several times over. Ansible is a great tool for that because every step is idempotent.

I would also do the migration on a staging server instead of the live server, if possible. Have the DNS TTL reduced to 300 seconds two weeks in advance. The migration would copy over the production site to staging (you can use Akeeba UNiTE to automate that) and perform the migration there. You can test the migration and then swap the DNS records so that production becomes staging and staging becomes production.

This gives you a lot of safety because you have a hot failsafe: if you missed something you can swap the servers again and even restore processed tickets by marking deleted emails as non-deleted and unread directly on the IMAP server. Moreover, it reduces the hard outage window. When you are about to start taking the backup you need to only disable the features which modify data (creating / editing user accounts, submitting / replying to tickets, etc); the site is still there. When the switchover happens everything is magically working again, on a brand new site. This minimises the perceived downtime. I've done that a number of times with this site. Our perceived downtime was about 5' to 10' each time with the actual time window when features were turned off being around 20' to 30'.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Wednesday, 02 August 2023 14:28 CDT

[email protected]

Nicholas,

It seems I have hit another roadblock....

My process is as follows:

1.) Setup ATS 5.2.8 including categories, custom fields, mail templates

2.) Alter the file uninstall.mysql.utf8.sql prior to uninstall. Removing sections that uninstall categories, fields, field groups, and mail templates.

Doing this preserves fields, field groups, and mail templates, however, ATS categories are still removed by the uninstall process.

3.) Run SQL statements using phpMyAdmin to re-add the ATS categories removed by the uninstall of ATS 5.2.8

4.) Install ATS 4.0.8

5.) Restore production J3/ATS4 tickets DB on new J4/ATS5 instance

6.) Install ATS 5.2.8

7.) Run SQL commands to change the production category ID's on existing tickets to match the new J4/ATS5 category ID's

At this point the data is migrated and we have moved the tickets to the new J4/ATS5 system.

Issues/Workarounds:

1.) I must go through the ATS categories and redo the ATS permissions for the "Support Staff" and the user groups that need to submit tickets. It seems the uninstall and reinstall process removes permissions and I must do this again. This is manageable.

2.) In my categories, I am using custom fields and conditional display or "Show On". After restoring and upgrading ATS the fields and conditional display work properly for a super user only. I have gone back to the fields and set permissions for the "Support Staff" and users submitting the tickets but this does not fix the conditional display for non "super users". Additionally, on the upgraded instance of ATS 5.2.8 the custom fields no longer have a "Show On" attribute shown. (See screenshots)

At this point...I am stuck and cannot move to our new system until I can resolve the custom fields issue. Any assistance or suggestions would be greatly appreciated.

Thanks in advance!

Eric

Wednesday, 02 August 2023 15:12 CDT

nicholas

It seems the uninstall and reinstall process removes permissions and I must do this againText

Correct. Permissions are stored in the #__assets table. The category asset records need to be under the component's asset record (based on the lft/rgt values) since permissions in Joomla! are hierarchical. I think the easiest way to handle that is to copy the rules field values into the new categories' asset records.

About the custom fields, make sure that Akeeba Ticket System's Content – Akeeba Ticket System Custom Fields Show–on Behavior plugin which enables the Show On feature is published.

Fields' Access is handled by Joomla itself, namely the content plugin for custom fields which calls code from the core com_fields component. I actually have no say in that. As a developer, all I can do is expect Joomla! to “magically” populate the jcfields property of my ticket table object as I explained in https://www.akeeba.com/documentation/ats-for-joomla/ticket-fields.html. I then just display the values based on the contents of that property, as you can see in the view templates. That's what even core code (e.g. com_content) does.

Also remember that you are reinstalling the component, therefore the Permissions for ATS for each user group are reset. See the first paragraph of this reply. You would need to either set up the permissions by hand on each group OR you would need to copy the rules field value for the #__assets table record which has name = com_ats and level = 1.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Wednesday, 02 August 2023 21:37 CDT

[email protected]

Nicholas,

Thanks for taking time to provide additional information.

I still don't understand why the restore process would prevent the "Show On" attribute from being displayed when viewing an ATS created custom field in the back end. I have verified the Content – Akeeba Ticket System Custom Fields Show–on Behavior plugin is published.

It is clear to me that uninstalling ATS 5.2.8 even with the modified sql uninstall script is removing other data dependencies that is likely causing issues that I could be tracking down for a while. What about a new strategy. After all we have a very handy backup and recovery tool for Joomla available.

On the new J4/ATS5 Server

1.) Take a Full DB only backup to capture all new ATS categories, fields, field groups and mail templates

2.) Uninstall ATS 5.2.8

3.) Install ATS 4.0.8

4.) Restore production J3/ATS4 ATS DB tables from most recent backup archive

5.) Install ATS 5.2.8 to upgrade DB tables and ATS to latest release

6.) Perform DB restore of all tables except #_ats_* from backup archive created in step 1

**At this point the new J4/ATS5 server is ready to be put into production as all previous ticket data has been migrated.

If my logic is correct, this would restore the J4/ATS5 DB tables to it's state prior to removal of ATS 5.2.8 in step 2 and leave the ticket data restored and in ATS5 format as this was done in step 5?

Worst Case Scenario on the new J4/ATS5 Server:

1.) Uninstall ATS 5.2.8

2.) Install ATS 4.0.8

3.) Restore production J3/ATS4 ATS DB tables from most recent backup archive

4.) Install ATS 5.2.8 to upgrade DB tables and ATS to latest release

5.) Manually create the ATS Categories, Fields, Field Groups (it seems the mail templates are working properly)

6.) Using SQL scripts manually update the catid of the existing tickets to match the ID's of the newly created ATS Categories

**This option is less desirable, due the time required to re-create the custom fields and show on behavior.

I welcome any feedback or suggestion on my new approach.

Thanks for your support!

Eric

Thursday, 03 August 2023 06:24 CDT

nicholas

I think it's best to eliminate the step where you uninstall ATS. I believe this is the step that wreaks havoc to the permissions. Reinstalling it creates a new asset ID for com_ats which I suspect is what causes all the problems you have.

Here's another idea:

Take a backup of the old site.
Restore it into a new site we will call temp.
Disable all third party extensions except for ATS 4 on temp.
Upgrade temp to Joomla 4.
Upgrade ATS from 4 to 5 on temp.
Use Akeeba Backup or phpMyAdmin to make a copy of the #__ats_tickets, #__ats_attachments, and #__ats_posts tables only.
Back on the development site, replace the #__ats_tickets, #__ats_attachments, and #__ats_posts tables from the backup you took on the previous step.
Copy media/com_ats/attachments from the temp site to the development site (these are the attachment files).

Assuming that your category IDs have not changed, this will transplant the old tickets with their posts and attachments to the dev site.

The idea here is that the less you need to tweak the (proven working) dev site, the less likely it is that something breaks. Does that approach work for you?

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Thursday, 03 August 2023 11:21 CDT

[email protected]

Nicholas,

Thanks for the suggestions, however, when I was trying to upgrade my current production site I kept running into so many issues with the Joomla 3 > 4 upgrade today. It was taking way too much effort, and the Joomla upgrade never completed successfully, which is why I originally started with a fresh install of Joomla when building our new site.

I went back and tried my steps, which when using Akeeba Backup, sure makes life easy to snapshot the entire system or DB, allowing me to rapidly test the process and return to a baseline quickly!

Here is a high-level overview of the process that seems to be working for me now. The information you provided was helpful to understand the category and field permission issues.

On the new J4/ATS5 Server

1.) Take a DB backup to capture all new ATS 5.2.8 categories, fields, field groups and mail templates, permissions, etc.

2.) Uninstall ATS 5.2.8

3.) Install ATS 4.0.8

4.) Import backup archive in Akeeba Backup and restore production J3/ATS4 ATS DB tables from most recent backup archive

5.) Install ATS 5.2.8 to upgrade DB tables and ATS to latest release

6.) Perform site DB restore of all tables from the backup created in step 1, except the following to preserve Ticket data restored from Production site:

gna_ats_tickets
gna_ats_posts
gna_ats_managernotes
gna_ats_attachments

7.) Copy the attachment images from Production site to New site

*This process appears to resolved the issue with the disappearing "Show on" attribute for ATS custom fields. In addition the new tickets conditional display for the fields works for all levels of user permissions, not just superusers.

**I can get the data migration process done in under 15 minutes so my outtage window is relatively small to do the cutover.

If you see anything I may have missed, or potential problems I would appreciate your feedback.

Thanks,

Eric

Thursday, 03 August 2023 11:47 CDT

nicholas

Nice job! I can't think of any issues off the top of my head. Do a few test upgrades on a staging server, testing the living heck out of it, just in case both of us missed something.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Thursday, 03 August 2023 15:24 CDT

[email protected]

Nicholas,

Update....the data migration process was mostly successful! The issues I have found are related to email notifications.

New Tickets: notifications are sent to the owner, and any support staff that is a Super User.

Update Ticket: same as above

Manager Notes: Sent to all "Support Staff" including those not site Super Users

Assigned Tickets: Notification does not get sent to non Super Users

I tried resetting the mail templates but this did not solve the issue. Do you have any suggestions?

Thanks,

Eric

Edited by [email protected] on 2023-08-03 15:26 CDT

Thursday, 03 August 2023 22:29 CDT

[email protected]

Nicholas,

Another update after testing email notifications on a fresh J4/ATS5.2.8 system. It seems the issue is not limited to a system that has undergone the data migration process.

Unfortunately, this is a show stopper and I have a commitment to cut over this weekend. The biggest issue is that email notifications are not being sent for new tickets and when a user updates their ticket. I think I can workaround this issue using some email groups and filters on a Super User mailbox. However, this would be a temporary solution and won't work long term.

As a reminder, we have the use case in which our category support staff Joomla groups do not have "Manager" as a parent. I tried adding a user to a Manager Group, then giving that group "Support Staff" permissions, but this did not resolve the email notification issues.

In addition I added a user to Super Users and this did not resolve the issue either. It seems that email is working for all scenarios only for the original Super User account.

The biggest issue is that user posts to a ticket will not trigger a notification.

Notify Assigned Manager = On in the component

Category is configured for all "Support Staff" to receive notifications

New Ticket Notifications:

- Submitting user receive email notification (expected)
- Support Staff who are super users receive email notification (expected)
- Support Staff who are NOT super users do NOT receive email notifications (unexpected)

Ticket Post Updated (Unassigned Ticket)

- Support Staff who are super users receive email notification (expected)
- Support Staff who are not super users do not receive email notifications (unexpected)

Ticket Post Updated (Assigned Ticket)

- (User post) Assigned Support Staff who are NOT super users do NOT receive email notification (unexpected)
- (User post) Assigned Support Staff who are Super Users receive email notification (expected)
- (Support Staff Post) User receives email notification (expected)

Assigned Ticket Notification

- Support Staff who are NOT super users receive email notification (expected)
- Support Staff who are Super Users receive email notification (expected)

Manager Note added (Assigned or Unassigned Ticket)

- Support Staff who are NOT super users receive email notification (expected)
- Support Staff who are Super Users do NOT receive email notification (unexpected)

Any assistance or suggestions would be appreciated.

Eric

Edited by [email protected] on 2023-08-03 22:32 CDT

Friday, 04 August 2023 02:01 CDT

nicholas

All of the users who do not receive email have "Receive System Emails" set to No in their user profile. ATS respects that and does not send them email.

Solution: Edit each of these users. In the Account Details tab for each user set "Receive System Emails" to Yes and click on Save & Close in the toolbar.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Friday, 04 August 2023 06:55 CDT

[email protected]

Nicholas,

This doesn't seem to the issue. Also if this was the case, why are some of the notifications going the user but not all?

Is this possibly related to the mail templates?

Is it possible for you to look at my dev server? I can provide credentials if this is a private ticket.

Thanks,

Eric

Friday, 04 August 2023 08:09 CDT

nicholas

I am testing on my local site. I have the following support staff:

alice, Super User Super User (implied support staff, does not belong in any other group). This is equivalent to your Super Users.
bob, Manager who also belongs to the Support Staff group.
charlie, Administrator who also belongs to the Support Staff group.
frontman, Registered who also belongs to the Support Staff group. This is equivalent to your use case.

I made sure all of them have Receive System Email set to Yes.

I also have the user client who is, well, the client submitting tickets.

I am using the default email templates. Note that if an email template is only available for a specific language (e.g. en-GB) and the client logged in uses a different language (e.g. en-US) then no email template is found and no email is sent. If a mail template is unpublished or missing, again, no email is sent. So, do check that your email templates are set up correctly.

I just ran the following tests on my dev site, all from the frontend of the site — though this doesn't matter because the emails don't check the logged in user, they operate based on the user who owns the ticket, the ticket who owns the new post, and whether the ticket is new or existing. That is to say, it does not matter if you create a ticket / post to a ticket / post a manager's note / otherwise manipulate a ticket from the frontend, backend, a CLI script, or even a custom solution, as long as everything goes through ATS' internal TicketTable and PostTable classes (which is what always happens in our code).

New Ticket (logged in as client): All support staff received their emails. The client received his new ticket email, too.

Reply to unassigned ticket (logged in as client): All support staff received their emails. The client does not receive an email, as expected.

Assign ticket (logged in as alice, assigning to frontman): frontman receives an email. Nobody else does (expected).

Reply to assigned ticket (logged in as alice, ticket assigned to frontman): All support staff received their emails. The client received his email too.

Reply to assigned ticket (logged in as frontman, ticket assigned to frontman): All support staff except frontman received their emails. The client received his email too.

Manager note (logged in as frontman): All support staff except frontman received their emails.

Manager note (logged in as alice): All support staff except alice received their emails.

So, it looks like ATS emails do work as intended.

Beyond Receive System Email there's one other thing which could prevent support staff from receiving email. In each Category, Ticket Options there are two options: Notify Support Staff and Exclude Support Staff.

If Notify Support Staff is set to All Support Staff then all people with Support Staff privileges in the category will be notified except those in the Exclude Support Staff list.

If Notify Support Staff is set to a list of users then only those users, except those in the Exclude Support Staff list, will be notified as long as they have Support Staff privileges in the category.

If things are getting weird, try setting Notify Support Staff to only All Support Staff and leave Exclude Support Staff empty in all of your categories.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Friday, 04 August 2023 08:14 CDT

nicholas

One further note. To test the email sending I use MailHog. This is deliberate. It tells me whether the email was sent, which is what I am trying to determine.

If I route the email through a mailserver I won't be testing that. I will be testing whether the email was sent, the mailserver accepted it without identifying it as spam (or otherwise reject it), managed to route it to the recipient, the recipient's mailserver acceptet it without identifying it as spam or otherwise misroute it, the mail client managed to connect to the recipient's mailbox and retrieved all messages, and the mail client did not itself categorise the mail message as spam or otherwise refrained from displaying it. There are so many moving parts I don't control that it does not even make for a remotely suitable test method. For example, I've seen too many mailservers where having the word "test" in the subject line causes the email to bounce.

When testing, check the logs of your SMTP server to see if an email message was delivered to it from a remote client that's your (development) site.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Friday, 04 August 2023 11:39 CDT

[email protected]

Nicholas,

I have configured our development server to use an SMTP relay which allows me to view the activity.

As you can see in the screenshot, the new ticket notification is sent only to the Super User "[email protected]" and no other "Support Staff". This is true for the scenarios I previously documented.

I have verified

All "Support Staff" have "receive system emails" enabled

The category is configured "Notify All Support"

Once again...some email notifications are being sent to non-Super User "Support Staff" but not the following ones. (Most Important for our use case)

-New Ticket Notifications

-Ticket Update Notifications

Here is a screen recording showing the category configuration:

https://watch.screencastify.com/v/HCTIrIl3mIsUtFXfC9uh

Attached is screenshot of our smtp queue showing that email notifications for the ticket did not get sent to all configured "Support Staff". It should have been sent to these additional staff:

[email protected]

Additionally, I can confirm that these accounts can access this ticket as "Support Staff" from the browser interface.

Please let me know if you have other suggestions to resolve this. Could there be an issue, similar to the previous bug we hit with mail fetch and accessing the categories?

Thanks,

Eric

Friday, 04 August 2023 13:24 CDT

[email protected]

Yes...this is the never ending ticket!

On a whim, I decided to try "Save and Copy" on the category in which some of the email notifications are not being sent.

Turns out that the new category that is a copy is working as expected in my preliminary tests.

I will update (if you are interested) if this resolved the email notification issue.

Out for now!

EJ

Friday, 04 August 2023 14:54 CDT

[email protected]

Unfortunately...this did not resolve the issue. I still have an issue with New Ticket Notifications and Ticket Update notifications.

My categories are all private I tried changing visibility to Public and re-testing. This did not make a difference so there is still an issue.

Thanks,

Eric

Edited by [email protected] on 2023-08-04 15:00 CDT

Saturday, 05 August 2023 09:45 CDT

nicholas

I see that your category has some excluded managers. I tried doing that too, in case there was some obscure bug I had not figured out yet, but the emails are still sent when doing the same checks. Also, as you said, private or public ticket (or the visibility of the category) matters not; it's not even a factor in the email sending.

Are you sure these two user accounts corresponding to the tech1 and tech2 email addresses can manage tickets in this category? Joomla's permissions are very powerful, but also convoluted, as there are two axes of hierarchy: the group hierarchy axis and the component-category-subcategory axis. Add to that the complexity of a hard Deny in any level of either hierarchy overriding any explicit Allow in any other levels of either hierarchy and you have a proper mindfsck. Please try logging in as these users and try managing some tickets in the tech support category to see if it works as expected.

Moreover, do check if the have Receive System Email set to Yes in their user account. I am not sure if you've checked that? You may have told me, but I can't find that information — as you said, it's a long ticket :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Tuesday, 08 August 2023 12:39 CDT

[email protected]

Nicholas,

Sorry for the delay in responding.

1.) Regarding the user properties "Receive System Emails". Turing this on only seems to affect New Ticket Posts and Ticket Updates. It didn't seem to affect other notifications and support staff would get manager note notifications, ticket assigned notifications, etc.

2.) I have set the ATS category notifications to "All Support Staff" and then excluded staff members or removed "All Support Staff" and specified staff members. In both cases all support staff members received notifications. It seems that ATS was ignoring this setting and notifying all Support Staff no matter what the setting.

I have gone through the permission structure for the component and the categories and there is nothing obvious to me.

At this point, my workaround is to create different categories based on notification requirements. We can re-assign the ticket to the appropriate category if needed. The added benefit of this approach is it better silo's support staff based on their area of expertise.

Ultimately, I understand that the restore process may have introduced some "strange behavior", but I can live with the system as it is.

Thank you for all of your patience and support getting us to ATS5 finally!

Akeeba Backup was the super star performer for us. It allowed me to quickly take system snapshots and return to a known good state during our testing.

Eric

Wednesday, 09 August 2023 11:55 CDT

nicholas

That's totally not what is happening on my test sites (plural; I've been testing today with two different sites, one hosted on macOS, the other on Ubuntu Server).

Same setup as last time with all these managers. The category is now set up with Notify support staff set to All Managers and Exclude support staff listing all managers except Charlie.

I tried creating a new ticket, and reply to a ticket as the client: only Charlie (and the client, for new ticket) gets an email. I tried replying as another manager: only Charlie gets the email.

I assigned the ticket to Bob. Bob gets an email telling him so.

I created a manager's note. All managers get the notification, which is intentional. The idea is that all managers of a category should be notified about manager notes as they may have information relevant to their own tickets with that client. <-- I think this may be confusing to you?

After the ticket was assigned, I replied to it both as a manager (namely, Alice). Only Charlie and the client got the email. Replying as the client only sends an email to Charlie (not Bob). This is also intentional. Ticket assignment is just a flag to say "this ticket should be handled by X person". It does not control email delivery. The category settings control email delivery. <-- I think this may be confusing to you?

The thing is, the use case for ATS has been software support in a relatively small shop, like ours. Trying to change that would make ATS not suitable for us. Considering that ATS does not even make anywhere near enough money to justify its development were it not for the fact we need it for our own site, if we change the way emails work it would make ATS a non-viable product.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Wednesday, 09 August 2023 12:44 CDT

[email protected]

Nicholas,

I appreciate your investigations and testing. I believe that I understand how the notifications should work and in fact have tested our notification scenarios early on in our build process.

I am finding that I am running into "unexpected" behavior on our current build. This is occurring before performing any data migration from J3/ATS4.

Is it possible that during our build process, that incrementally updating from ATS 5.0.x including multiple developmental builds may have introduced some of these odd behaviors? If I do a fresh install of Joomla and ATS 5.2.8 I do not get any of these behaviors. This seems to led me down the path that all of the incremental updates may be the root cause of these issues.I have submitted two new tickets regarding email notifications

mailfetch is ignoring "Reply by Email" = No.
404 Error when user clicks URL link in Manager Note email notification

Davide has requested a site backup, which I plan to provide shortly for investigation. Concurrently, I can try removing ATS 5.2.8 completely on a test server and then do a fresh install of 5.2.8 and see if I still get this odd behavior with email notifications.

Do you have any suggested process to ensure removing ATS 5.2.8 will remove all remnants of any previous versions?

thanks,

Eric

Thursday, 10 August 2023 04:10 CDT

nicholas

Uninstalling ATS will cause problems since it removes the data from the database. It also doesn't make sense as a solution. ATS 5 has its files in the src directory, ATS 4 has had them in directories directly under its main folder — they are using different MVC frameworks.

What would make more sense is that there is a mix of old and new versions of ATS, either because of the backup restorations, or because Joomla sometimes fails to copy files on update. Both can be fixed very easily. Install the latest version of ATS 5 twice in a row. This will definitely copy the correct files in place (or give you an error about files not being copied).

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support