Monthly Archives - March 2020

How we hosted a big sale on Magento without a hitch

During an online sale that includes promotion to a list, brands worry about downtime due to “unexpected high load”. We believe that while every system has a rated limit, there can be flexibility added to your infrastructure. With some planning, including autoscale and tuning, a successful sale event can be a very likely outcome. See how we handled one such sale without a hitch.

In our experience promotions via SMS generates very high peaks. In this article we will show our recent experience with a sale day that generated a peak of 30 hits per second to Magento.

external

The sale

On Saturday 21st March, 2020 one of our customers had a sale event. The merchant – a fashion retail brand - is an omni channel retailer in India with about 145 stores across 45 cities. There have been previous sale events, but this one was different as most stores were closed due to the corona virus measures in place. This meant that more customers were likely to shop online than visit the store.

The sale was promoted via sms and email. The SMS list was about 1 million. Promotional messages were sent in batches of 200k every half hour. Marketing was to start at 10 AM local time.

Monitoring the website

We monitor the nginx log file, analyze and display information on our dashboard. A key component of our dashboard is the plot of number of hits / minute (we count only php hits, static hits like css, js and images are excluded). We also calculate avg response time per minute as well as standard deviation per minute. Both of these have a unit of seconds. We also have alerts setup for many parameters including slowdown and a 5xx error response.

The graph below is for a 24 hour period, including the sale time, which lasted about 5 hours with the effect keeping the site more busy than usual afterwards. Note these are server logs graphs – they look quite different from google analytics!

The traffic & response

The graph gives as an idea of the traffic to the website. The turquoise line is hits per minute served by php.

  • Traffic peaked a about 2000 hits per minute – about 30 hits per second.
  • 3476 items were added to cart during the sale.
  • Average server response was mostly below 1 second.
  • There were 3 errors – all of them related to checkout function (“This payment method is not available”)

Mysql query cache and response time

The store is a Magento 1 website. Just as the traffic / hits to the website started peaking, we got a slow alert – represented by the large black line close to the first traffic peak around 10.45 am in the previous figure. Our prior experience with the website showed that the mysql query cache plays a role in the site performance. Before the sale started, mysql query cache was turned off. On seeing the slowdown alert, we turned on the mysql query cache and saw an immediate improvement in performance. Both avg response per minute and standard deviation per minute improved – for some time.

However, standard deviation – higher number indicates some visitors faced slower response to the website than others – deteriorated with time. After the sale (10pm local time), mysql query cache was turned off – resulting in improved standard deviation, but slightly worse avg response time.

AWS autoscale

The following graph plots AWS autoscale instances over the same 24 hour period. A time based minimum 5 instances was planned during the sale from 9.15 AM to 7.30PM. At the peak 7 instances were required – our autoscale policy adds 2 instances and removes 1 at a time.

luroConnect Architecture for AWS and controls

Our standard scalable architecture uses nginx as a load balancer. The communication between the load balancer to php servers is using fcgi – individual php servers do not have nginx. A NFS server is used to share folders that need sharing – media and var folders in Magento 1. Magento cache and sessions are served from redis. Code is also shared using nfs.

A custom lambda function communicates between autoscale lifecycle hooks and the nginx load balancer adding and remove instances as the AWS autoscale policy decides.

Our cache controls including the ability to turn on and off mysql cache, resize memory allocated to redis cache, resize memory allocated to php-fpm opcache.

Interested in knowing about the advanced architecture of luroConnect?

Fill the form below and we will contact you.

    My company owns the Magento site Yes, I am a developer I represent a Magento Agency
    March 25, 2020By Uncategorized0 CommentsRead more...

    Magento emails & SMTP

    Emails are a very critical portion of an eCommerce website. There are 2 types of emails that are used – transactional and promotional.

    Transactional emails are emails the eCommerce system sends for regular customer activity – such as order confirmation email. It may also include abandoned cart email, wish list reminders, password change, etc. If a website visitor action causes an email to go out, it is a transactional email.

    In this article we will discuss the best way to configure transactional email sending in Magento. This article applies to both Magento 1 and Magento 2. The focus is on what happens on production systems.

    The SMTP Plugin

    Traditionally Magento agencies and developers install a “SMTP Plugin”. A simple search in Magento marketplace for SMTP reveals many options. Some are general plugins that allow you to connect to any SMTP provider, others are from vendors such as SendGrid.

    Advantages of such plugins are

    • Easy setup
    • Developer testing is easy
    • Websites on shared hosting have to use such a plugin

    However, for growing websites, this is a particular problem

    • Outgoing ports from the application server may not be open
    • Many SMTP providers have restrictions such as rate limit on email sending or number of connections. Some SMTP plugins may not handle this condition well - just reporting the error vs trying to send again.
    • They may not be able to alert when email sending is down
    • When upstream providers' SMTP service is down or slow, cron will slow down as each email times out on sending. (Magento sends emails in cron).

    Using postfix

    By default, Magento sends emails to the local system where php is running. Linux systems have many options for receiving such requests and sending out the emails. A popular software is postfix. While postfix can directly send emails to the recipients’ email systems, we are interested in the “relay host” mode of postfix. In this mode, postfix acts to relay emails sent to it from Magento to the upstream provider – such as SendGrid.

    Advantages of postfix

    • Postfix has an inbuilt queue. When the upstream server’s rate limit is reached, a “defer” status is returned. Postfix understands this and it will queue deferred emails and keep retrying. Emails are not lost.
    • Magento cron will deliver to local postfix in a matter of milliseconds, freeing up cron to perform other activities.
    • Connections can be pooled so if many emails are in queue they will be pooled together and sent using a single connection. Some SMTP servers, notably gmail, have a rate limit on the number of connections.
    • Log files that can be checked by system admins. Log files can also be sent to a log collection tools. Advanced commands help checking the queue, even trimming it using conditions such as email subject.
    • Flexible configuration allows many features – on a staging server for example, only whitelisted recipients can receive emails, preventing accidental trigger of email to customers. Or emails to known spammers can be filtered by email id or domain.

    Installation and configuration

    Installing postfix is easy – the package is part of any linux system. For RHEL based system like CentOS :

    sudo yum -y install postfix
sudo yum -y install cyrus-sasl-plain cyrus-sasl-md5

    Configuring postfix to act as a relayhost.

    Click on the tab below to see configuration information for the specific SMTP service being used. Commands below should be run as root (or sudo). Ownership of files created, especially the password files is assumed to be root.

    Sendgrid

    1. Sign in to SendGrid and go to Settings > API Keys.
    2. Create an API key.
    3. Select the permissions for the key. At a minimum, the key must have Mail sendpermissions to send email.
    4. Click Save to create the key. SendGrid generates a new key. This is the only copy of the key, so make sure that you copy the key to clipboard and use if for the next step.
    5. Create the hashmap from the key 
      echo [smtp.sendgrid.net]:2525 apikey:{{{PASTE}}} >> /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
rm /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd.db
    6. Add to postfix configuration file main.cf 
      cat >> /etc/postfix/main.cf <<EOF
smtpd_tls_security_level = may
# enable SASL authentication
smtp_sasl_auth_enable = yes
# tell Postfix where the credentials are stored
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
header_size_limit = 4096000
relayhost = [smtp.sendgrid.net]:587
smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
EOF
    7. service postfix restart

    Mailgun

    1. Signup for maligun account.
    2. Get credentials for maligun. Mailgun has an option for SMTP username and password under the Domains section of the mailgun portal.
    3. Create the hashmap from the key 
      echo [smtp.mailgun.org]:2525 {{user}}:{{password}} >> /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
rm /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd.db
    4. Add to postfix configuration file main.cf 
      cat >> /etc/postfix/main.cf <<EOF
relayhost = [smtp.mailgun.org]:2525
smtp_tls_security_level = encrypt
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
EOF
    5. service postfix restart

    Amazon SES

    Amazon SES is the most cost effective SMTP provider to our knowledge. However, it is also the most difficult to setup as it requires you to verify sender emails and sandbox mode (which we will not consider here).

    1. Login to your aws account
    2. Select “Simple Email Service” from the services
    3. Select Domains from the left menu
    4. Click “verify a new domain”
    5. Enter domain and click on verify
      SES will generate the TXT records you need to add to your DNS

      SES will also generate the DKIM CNAME record to be added to your DNS
    6. Ensure domain verification green status
    7. In the SES main screen, select “Email Addresses” from the left menu
    8. Enter the sender email address. Sender email address are set in Magento. Depending on your Magento configuration different emails may need to be configured. Each email needs a mailbox as a confirmation email is sent to it.
    9. Refer https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html to create SMTP credentials. Note username and password.
    10. Depending on the region your SMTP endpoint (relayhost setting in the main.cf below) will be different. Refer https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-connect.html
      Note : The region of SES is unimportant in that there are no restrictions from where the email is sent. Your Magento instance can be anywhere - not even on AWS.
    11. Save password 
      echo [email-smtp.us-west-2.amazonaws.com]:587 USERNAME:PASSWORD >> /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
rm /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd.db
    12. Update main.cf 
      cat >> /etc/postfix/main.cf <<EOF
relayhost = [email-smtp.us-west-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
EOF

    Advanced configurations

    Postfix relay gateway

    In a more mature environment, each php server may not be able to send on SMTP ports. Even in the presense of a NAT gateway, we would recommend a main postfix gateway to ease logging and prevent loss of emails when an instance is lost in autoscaling. With postfix we can create a relay – php servers are configured to send email to a designated email gateway server which in turn relays to the SMTP upstream server.

    Sending promotional emails

    Promotional emails should ideally not originate from your website – instead they should be from your marketing tools. However, Magento does support sending promotional emails. To improve deliverability of crucial transactional emails, we recommend using a different sender email domain for promotional emails. If this is configured in Magento, postfix can handle sending promotional emails from a upstream service different from the transactional service. Assuming username / passwords have been created in the SMTP service for the promotional domain, the following configuration allows selecting the relayhost to use based on the sending domain.

    cat << /etc/postfix/main.cf >>EOF
smtp_sender_dependent_authentication = yes
sender_dependent_relayhost_maps = hash:/etc/postfix/relayhost_map
EOF

cat << /etc/postfix/relayhost_map >>EOF
user@promodomain  [smtp.xxx.com]:{{port}}
EOF

    Add the new domains passwords as the first line in /etc/postfix/sasl_passwd

    user@promodomain  {{user}}:{{password}}

    luroConnect and Magento emails

    The configurations described above are abridged versions of what is used in luroConnect. We use the advanced configuration with a designated server allowed to communicate on port 587 to the outside. We also use secure keys to ensure passwords are not saved in chef cookbooks / configurations. Instead they are stored in a secure key store.

    Interested in knowing about the advanced architecture of luroConnect?

    Fill the form below and we will contact you.

      My company owns the Magento site Yes, I am a developer I represent a Magento Agency

      Reactions to this post on linkedin

      March 20, 2020By hosting, Linux System Administration, Magento, performance, Site Speed0 CommentsRead more...

      Magento 1 end of life

      June 2020 is a few weeks away and many stores are not ready to move away from Magento 1. But wait, you don't have to update - now or in the long run.

      COVID-19 Update : We find many customers have had to delay their Magento 2 launch in these uncertain times. We also know many of them did not have plans to keep Magento 1 uptodate. In fact, we know agencies that have stopped support for Magento 1.

      Starting at a low cost of $200 per month with no long term contract. It includes reviewing your current hosting for security, moving your website to the latest Magento 1.9 and latest php supported as well as adding additional security measures to your website. It also includes help signing you up for Mage One or Open Mage projects for support beyond Magento 1 EOL, if required.

      Signup now (no credit card required) and we will be in touch with you.

      What does end-of-life for Magento 1 mean?

      Magento 1 End-of-Life does not mean your website will stop working. It means Adobe will stop giving fixes for Magento 1, even security patches. As php version in use goes out-of-life, no upgrades will be given by Adobe.

      However, being an open source platform, your Magento 1 website will not stop working. The code and license do not restrict you from running the website.

      Stay on Magento 1 for short or even long term

      That is a valid option and many customers are choosing this. Makes sense if

      • You have a lot of investment in the customizations which may be difficult to replicate anywhere
      • You have a stable money generating store and any change looks like a risk
      • Are in the process of migration, but the migration may take some time

      What are the options to stay on Magento 1?

      • Use paid support plan from Mage one (https://mage-one.com).
        luroConnect is a partner and we will apply the patches for you as they are released.
      • Use open source Magento 1 fork (https://github.com/OpenMage/magento-lts) with support from the community.

      What are the risks?

      • Support from either of the above reduces over time as many websites move out of Magento 1
      • Developer support may reduce as most developers move to Magento 2
      • Plugin vendors have already stopped support or are stopping support.

      Magento 1 and PCI

      Many merchants received email from or read their post and advice move to another platform after June 2020.

      It refers to PCI / DSS Requirement 6 - excerpted here with highlighting for relevance of discussion.

      Your Magento 1 store software has many vendors, including Adobe / Magento for the core, but also plugin vendors. Since it is open source, you are free to modify the core and take the responsibility and other requirements may apply.

      By switching to an alternative vendor for the Magento 1 core - such as Mage-One or Open Mage, in our non-legal opinion, you are not on Magento 1 any more and do not have Adobe as your vendor. If a plugin vendor does not give security patches to your Magento 1 plugin any more, it is important to take over the plugin code responsibility as a separate contract.

      PCI does not have a vendor approval process. However, your vendor may need to justify satisfying some other requirements for safe and secure coding practices.

      However, by not recognizing your core application, you may need to talk to PayPal as a merchant to get PCI approval. This would include scans.

      luroConnect Support for Magento 1 past EOL

      We have built add-on package for Magento 1 EOL support. We have a 4-point plan to support you.

      From EUR 29

      Mage-One Patches for post EOL Magento 1 support. Will satisfy PCI vendor requirement.

      mage-one logo

      From EUR 99

      Sansec eComscan examines a store for malware, vulnerabilities and unauthorized accounts. Written by well known security expert Willien De Groot, it scans files, databases and 3rd party components of Magento.

      external

      From USD 50*

      Inbuilt into our Nginx, with M1 rules, protects from OWASP Top 10, with the ability of virtual patching.

      From USD 50*

      Staging environment to ensure patches are tested before taken live.

      * This is in addition to our fees for WAF and staging environments if not included in your support plan. Customer pays for hosting costs of staging server.

      How we protect you

      1. File system security to prevent 0-day or new unknown vulnerabilities. Strict file and folder permissions prevent uploads to folders that execute code
      2. Support for Magento 1 Nginx rules not allowing execution of php from skin and  js or php from media. This rule will prevent many malicious code to fail as they depend the ability to upload malicious code and execute.
      3. WAF - Web Application Firewall - with strict Magento 1 rules. This prevents SQL Injection and cross site scripting related attacks from being allowed.
      4. Virtual patching - block URLs that are known to have vulnerabilities. For example, we do not allow saving of the "miscellaneous" header and footer section from being written from the admin login.
      5. Partnership with mage-one to get the latest patches and keep your site uptodate.
      6. Admin login protection via dual password. The first is a basic http challenge. This reduces password guess of the admin URL as 2 passwords have to be guessed.
      7. Password guess prevention by restricting how many failed attempts are allowed in a day from the same IP - implemented at the application server level without changing Magento code.
      8. Staging environment to test patches from open mage or mage one or any other source you may have. Also support php version upgrade first on staging before upgrading production.
      9. Protect source code by using secure deploy process
      10. Secure backup With a proven restore strategy
      11. Support for our secure deploy process that ensures 0 downtime during code deploy and not have git folder in the hosting folder. An ability to rollback by switching to any previous deployed version is an added advantage.
      12. System components upgrade - including php. As versions of php approach their security end-of-life and support for higher versions appear in patches, php version will also be upgraded.
      13. Partnership with Sansec for their eCommscan scanning product.

      We can analyze your site for free

      Schedule a call

      Not happy with your website performance and want an expert to look at it?

      • We will analyze your site using public information.
      • We will ask you to give us a 1 day web server log file.
      • We will try to identify what steps if any you should take to improve your sites performance goals.
      Click here to schedule the call

      Hosting help moving to Magento 2

      When moving to Magento 2, to reduce the downtime during the move, luroConnect has plans for you.

      • Staging server support plans.
      • Magento 2 transition plan with minimum downtime. Our care even includes URL rewrite rules to ensure SEO value is not lost during transition.
      March 6, 2020By hosting, Magento, Security0 CommentsRead more...