Batch Import Issues (August 2021)

Batch Issue Overview

In the first half of 2021 we identified an inconsistent, but not infrequent, issue with the roundtrip batch import process.

Affected sites are not successfully running batch imports. The issue can be identified by the following behaviors:

• No post-import success/error message is displayed (the usual green or red messages indicating how many nodes/atoms were created or updated and/or specifying errors).
• This can make it seem like the import is running successfully, but if no message is displayed, assume that it was not completed. The correct message is shown here.
• This NOT the AJAX message shown here. You need to click through that message to view the success/error message.
• The number varies – some sites will process one or two items, others 10 or 20.
• Checking the recent content page is a good way to see how many items were imported, based on the “Updated” timestamp field.
• Some, but not all items in the import spreadsheet *are* successfully imported. Go to /admin/content eg: https://mysiteURL.com/admin/content

The issue is not present on all Mukurtu sites. It seems to arise with certain hosting and database configurations and is related to Drupal feeds but is not consistent enough for us to pinpoint the exact cause. It has been observed in at least two hosting environments.

Our development team (lead developer, technical director, and support specialists) have spent several months trying to pinpoint and address the issue, but we have not been able to identify the root cause.

Our small team is currently focusing the majority of our development time and effort on the upcoming Drupal 8 based Mukurtu CMS 4 release. It is essential that this work be completed before Drupal 7 is no longer supported, so we can ensure that all users are able to migrate to new sites.

We cannot continue investigating the root cause of this issue at this time. Once Mukurtu CMS 4 is launched we anticipate the issue will be solved as we will be using a different import method.

With all of this in mind, it is important to note that effected sites are otherwise fully functional.

• Manually creating items through the web interface (one at a time) is unaffected.
• Smaller batches can be imported. There is a documented workflow (see steps below) that will allow you to keep using the batch import tools. Depending on how many items a given site will process at a time, this will have varying levels of efficiency.

We understand that this is not ideal. Our support specialists will continue to help with all other aspects of Mukurtu, so that you can continue your work as efficiently as possible. And of course, when Mukurtu 4 is available, we will work with you to migrate to the new version and ensure that you can batch upload as anticipated without issue.

Our thanks to the users who helped to identify and troubleshoot these issues. Your patience and support are greatly appreciated.

Resources Note (added September 10, 2021)

Sites with the ability to increase server resources may be able to lessen, or avoid, the issues described above. This. is *not* guaranteed to work, but has been effective on some sites. Of note, this will likely not have an effect on basic Reclaim Shared Hosting sites, more server resources are needed.

There are two components to this, both of which should be implemented:

1) Increase the PHP memory_limit settings.

Our default minimum recommendation has been 196MB, so consider increasing beyond that. We don’t have a specific recommendation as every site and install varies – you may need to increase it several times until you find a setting that works for your site.

2) Reduce the feeds process limit

As a drupal admin, go to: /devel/php

Paste and execute the following code: variable_set(‘feeds_process_limit’, 10);

Alternate Workflow 

Available as a PDF download.

1) Prepare your complete import spreadsheet as normal (see here for more information). 

2) Make a copy of this spreadsheet.  

a) You will be editing the copy, so it is important to keep the complete original spreadsheet separate. 

3) If possible, download the spreadsheet copy and work in a desktop spreadsheet tool (see here for recommendations).  

a) These screenshots were taken using LibreOffice on a Mac. 

b) This can be done in Google Sheets, it is just less efficient as you will need to download the spreadsheet multiple times. 

4) Import the spreadsheet copy as normal. 

a) If an AJAX message is displayed, ignore it and click through to the actual confirmation message (see here for more information). 

5) If a confirmation message is displayed (either immediately, or after following the link above the AJAX message), the import ran successfully. 

a) If the message is all successes (X nodes created/updated), no action is needed. 

b) If the message contains errors, this is likely unrelated to the batch import issue described here. Resolve them as needed and re-import. 

6) If you are redirected to the import page and no confirmation message is displayed, the import did not run successfully. Most likely some, but not all of the items were imported. 

 7) Go to the dashboard and view the complete recent content list by clicking “more” at the bottom of the box (or go directly to /admin/content). 

 8) Filter the list to only show digital heritage (or the appropriate content type if you were importing something else).  

 9) Check the timestamp to see how many items were imported in the most recent batch.  

a) In this case, 19 items were imported (in most cases it is less than that).  

b) In all documented cases, the items were imported in the order listed in the spreadsheet, but you should confirm which items were imported. 

10) Open the spreadsheet copy and identify the items that were successfully imported. Delete them from the spreadsheet. 

 11) Save the changes to the spreadsheet copy. 

a) If using Google Sheets, export the edited spreadsheet copy. 

12) Repeat steps 4-9 as many times as needed to import all items in the spreadsheet copy.  

a) The final import will usually display a confirmation message. 

b) In this case, only 2 imports were needed since the first batch imported 19 items, and the second imported the remaining 11 items. If your site is only accepting a few items at a time, you may need to run many batches.