Announcement

Collapse
No announcement yet.

Recurring Issue- Being investigated

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Recurring Issue- Being investigated

    Date: March 1, 2021
    Time: 10:15 a.m. ET

    We are investigating an issue with the recurring transactions that were done on February 28, 2021. There appears to be an issue with communication with our gateways which is resulting in inconsistent data and processing.

    We apologize for the inconvenience and are investigating the issue.

    Updates will be posted in this thread.
    Regards,
    Click & Pledge Support Department

    On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

    Join us @ the educational webinars: https://clickandpledge.com/webinars/
    Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
    Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

  • #2
    Status Update @ 12:25 p.m. ET

    We have identified the issue and are pushing the update on all servers. The following is the details of the issue:
    • Patrons were ONLY charged one time. The issue has not affected any donor as all transactions are posted to the gateways correctly.
    • The issue is related to "some" of the RECURRING transactions. Emphasis on "some" as only a fraction of recurring transactions have had the issue.
    • Transactions done on regular forms were not affected.
    • The problem is a reporting issue and not actually payment related.
    Today's running of the recurring transactions has been delayed to ensure the issue has been resolved. Once the update has been replicated and pushed to production we will run the recurring transactions.

    Sorry for the inconvenience.
    Regards,
    Click & Pledge Support Department

    On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

    Join us @ the educational webinars: https://clickandpledge.com/webinars/
    Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
    Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

    Comment


    • #3
      Status Update @ 4:47 p.m. ET

      The recurring problem has been resolved. The following provide insight into the status:
      • The recurring transactions for March 1, 2021 are currently being processed. We expect the recurring to complete around 10 p.m. ET.
      • Recurring transactions that ran on February 28 are currently NOT showing correctly in CONNECT.
      • Recurring transactions that posted to Salesforce have posted as one time transaction and NOT recurring.
      • We are currently working on resolving the remaining issues and will provide a fix for posted transactions and the CONNECT status by tomorrow, March 2.
      The team is currently working on testing the fix but given it involves record changes a lot of test need to be completed prior to pushing the fix to the production database.

      Sorry for the massive inconvenience. Of the 25K transactions that posted yesterday, 1,023 transactions generated an error due to an extremely strange bug. We are doing all in our power to resolve this issue in a timely manner.

      Once again we are sorry for any inconvenience this may cause.
      Regards,
      Click & Pledge Support Department

      On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

      Join us @ the educational webinars: https://clickandpledge.com/webinars/
      Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
      Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

      Comment


      • #4
        Date: March 2, 2021
        Time: 2:45 p.m. ET

        Status Update:


        Our team has been working on ensuring the update resolves the Causes that were affected by this issue. The following summarizes the progress to-date:
        • 493 Organizations were affected by this issue.
        • All data has been cleaned and updated in the primary database and currently being replicated. This update resolves and will show the correct data in CONNECT platform.
        • Salesforce posts still need to be cleaned and resolved. To fix the Salesforce data we have to post all transactions back to Salesforce. The repost will delete the prior post and post the correct data back to the instance.
        We have spent a great deal of time to ensure the data integrity is maintained and Salesforce update is handled with care. This is extremely critical since the autoresponders and internal triggers need to be monitored.

        Our marketing team is currently working on a letter to be sent to all affected organizations with all the steps being taken to resolve this issue in their Salesforce instance. Organizations that are not in Salesforce should consider the issue resolved as all data in CONNECT are currently correct.
        Last edited by CnP.Support; 03-03-2021, 07:10 AM.
        Regards,
        Click & Pledge Support Department

        On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

        Join us @ the educational webinars: https://clickandpledge.com/webinars/
        Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
        Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

        Comment


        • #5
          Date: March 3, 2021
          Time: 8:40 a.m. ET

          Status Update:
          • All Causes affected by this issue were notified yesterday at 4:00 p.m. ET.
          • Transactions in CONNECT are all fixed so Causes that are not using Salesforce are all taken care of and nothing else needs to be done.
          • The re-post to Salesforce is currently under test and will be applied tomorrow to allow time for updating old instances to the new release.
          • What will happen in Salesforce once the re-post happens:
            • The new corrected C&P Data will post to Salesforce
            • All Salesforce instances using the latest version will behave correctly with the new posted data. The previous C&P Data will be deleted automatically and the new data will have the correct information.
            • The posted data will post with Autoresponder set to OFF. This means no autoresponder will be sent with the new data. This is to ensure we do not cause confusion with the patrons.
            • All previously created records will be deleted:
              • Important point about Alias: The existing version does not delete the alias record if new data posts. An update is being tested where the alias record will be deleted. Once this feature has been added the instances will be updated.
          We ask that all clients request an upgrade to the new release, which allows for deletion of Alias record with a re-post. Please post the upgrade request to the forum: https://forums.clickandpledge.com/fo...ade-assistance

          As we have stated several times in the past, we are truly sorry for the inconvenience. This issue totally caught us by surprise and took us a while to figure out what the root cause is. Once we have updated everyone and the dust has settled we will post an explanation of what happened for everyone's amusement. One has to love the weird world of software..
          Regards,
          Click & Pledge Support Department

          On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

          Join us @ the educational webinars: https://clickandpledge.com/webinars/
          Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
          Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

          Comment


          • #6
            Date: March 3, 2021
            Time: 10:30 a.m. ET

            Status Update:


            We are working on a release of Salesforce PaaS to delete the alias record created by a transaction, if that transaction is deleted. In other words, if the C&P Data is deleted the Alias created by that Order number should also be deleted. In the existing release all footprint of the Order Number is being deleted except the Alias record. We think deleting the Alias as part of removing the transaction's fingerprint is justified and reasonable.

            The update will be available and released tomorrow, March 4. The changes are already made and the testing team is testing all scenarios to make sure alias removal is done correctly.

            Causes that were affected by this unfortunate event have 2 options as listed below:

            ++++++++++++++++++++++++++++
            Option 1:

            If you are NOT in a rush to have the transactions resolved in Salesforce then do nothing and we will take care of it all for you. Simply grant us login access and request an upgrade. Once your instance has been upgraded we will push the data and fix all issues.

            Option 2:

            If you ARE in a rush and want to have your data fixed NOW, then you may go ahead and post the data from CONNECT to your Salesforce instance manually. The data in CONNECT are correct and will post to your Salesforce instance resolving the issue. REMEMBER: The alias record will not be deleted so you need to delete the incorrect alias manually.
            ++++++++++++++++++++++++++++

            The new Salesforce update will be made available on Thursday March 4 and all new upgrade requests will be upgraded to this version.

            Since we would like to give everyone enough time to request upgrade we will post the data back to all instances on Monday, March 8, 2021. This should give everyone enough time to request the update and for us to ensure all instances are set up for the automatic post.

            If you have any questions please contact the support department or post it to the appropriate Forum channel.
            Regards,
            Click & Pledge Support Department

            On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

            Join us @ the educational webinars: https://clickandpledge.com/webinars/
            Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
            Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

            Comment


            • #7
              Date: March 4, 2021
              Time: 10:46 a.m.

              Status Update:
              • An update to the Salesforce PaaS application has been released. Causes that have asked for the update to their Salesforce instance are currently being updated.
              • Once the update has been completed we will push the data back to the Salesforce for all the transactions with invalid recurring information.
              • In this update, once a transaction is deleted the associated Alias record will also be deleted.
              Please submit the upgrade request to the forum so we ensure your instance is up-to-date.
              Regards,
              Click & Pledge Support Department

              On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

              Join us @ the educational webinars: https://clickandpledge.com/webinars/
              Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
              Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

              Comment


              • #8
                Date: March 8, 2021

                Status Update:
                • An email is being sent shortly to all affected organization notifying them of the upcoming post.
                • Updated data will post to all affected instances at 3:00 p.m. ET. We are pushing the time to late afternoon to ensure all Causes that have not yet asked to be upgraded have time to request the upgrade. PLEASE submit your upgrade request if you are not on the latest release of PaaS: Version 9.2103250011 [Release Date: 3/4/2021]
                • If your instance is NOT up-to-date please submit an upgrade request: https://forums.clickandpledge.com/fo...ade-assistance
                • Upon completion of the post, another email will be sent to all affected organizations notifying them of the completion of the post.
                • Please contact support if you see inconsistency in your data after the 3:00 p.m. post. Our team will ensure all issues are resolved to your satisfaction.
                Once the data is pushed to the instance the following steps will be followed:
                • Matching C&P Order Number in the instance will be DELETED
                • The new C&P Data will post with "Send Autoresponder" flag set to false. The new data will NOT generate a new autoresponder to ensure patrons are not confused with a dated email.
                • The entire footprint of the prior post will be DELETED: C&P Transaction, Opportunities, and ALIAS records if the instance has been updated. If the instance is not with the latest release the ALIAS record will NOT be deleted and will need to be deleted manually.
                • The Autoresponder log records will NOT be deleted. We have decided not to DELETE any autoresponder logs as the record shows an action that has already taken place and it is good information to have. Deleting cannot possibly remove the fact that the email has already been sent. Important point to keep in mind is since the C&P Transaction that has created that autoresponder is not available, after deletion, the Order Number hyperlink will be removed as that Autoresponder is not related to any corresponding C&P Transaction Id.
                A new field for the "Order Number": has been added to the C&P Autoresponder log record to record the order number that had trigged the email. Prior to this release the relationship was set to the C&P Transaction but since that record is now deleted we need to make sure we can reference the record in the future. This feature was already added to the Release 10, but was moved to this release. The post-installation script will auto-fill the Order Number to all the legacy record to ensure data consistency in future. This post-installation script will ensure backward compatibility.

                Please make sure your instance is up-to-date prior to 3:00 p.m. ET.

                Once again, we apologize for the inconvenience and our team will do whatever it takes to ensure your data consistency is maintained.
                Last edited by CnP.Support; 03-08-2021, 08:23 AM.
                Regards,
                Click & Pledge Support Department

                On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

                Join us @ the educational webinars: https://clickandpledge.com/webinars/
                Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
                Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

                Comment


                • #9
                  March 8, 2021
                  Time: 3:00 p.m. ET

                  Status Update:


                  We started receiving upgrade requests after 2 p.m. ET and have not yet finished upgrading all the "last minute" requests. We truly are trying to help and accommodate all clients but waiting till the last moment is going to cause a bit of delay.

                  https://forums.clickandpledge.com/fo...ade-assistance

                  Sorry for the delay but we are working hard to upgrade all the last minute requests. As such the data posting may be delayed a bit.

                  We will update the post once the data posting starts.
                  Regards,
                  Click & Pledge Support Department

                  On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

                  Join us @ the educational webinars: https://clickandpledge.com/webinars/
                  Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
                  Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

                  Comment


                  • #10
                    March 8, 2021
                    Time: 6:00 p.m. ET

                    Status Update:


                    We just finished pushing all the transactions with the recurring issue back to Salesforce instances that were affected. Sorry for the delay in the posting from the original time of 3:00 p.m. We had to update all last minute requests for upgrade before posting data. A few requests came at 3:30 p.m. while other upgrades were in progress.

                    Data posts started after all the requests were upgraded.

                    Please review your instance and if you are seeing any issues please grant us login access and post your Salesforce Org ID with the detailed information about the issue as well as the Order Number. Our team will review and fix any remaining issues.

                    Questions? please let us know.

                    & once again, as we have said over and over again, we apologize for the inconvenience.
                    Regards,
                    Click & Pledge Support Department

                    On Salesforce? Help us by rating our app: Click & Pledge Donor Management on AppExchange

                    Join us @ the educational webinars: https://clickandpledge.com/webinars/
                    Live Support available Join between 3:00 - 3:30 p.m. ET Monday - Thursday: https://clickandpledge.com/webinars/
                    Are you on Salesforce? Join us at the Power of Us Hub: https://powerofus.force.com/0F980000000CjpC

                    Comment

                    Working...
                    X