Magento

luroConnect Image Optimization Support : Watchimage

Image Optimization is a basic requirement to have a faster website. Photo images used in websites are generated with too large a size to be served over the internet on a web page. Also browser support for different, more modern and efficient (by storage) formats vary, requiring an image optimizer to be used on a website.

Generating images in different formats may be required in websites to support different browsers. This is part of image optimization.

luroConnect WatchImage service, “watches” a set of directories where images may be generated or saved and automatically generates different formats for the same image – keeping the base file the same and renaming the extension appropriately.

For example for Magento, if one wants to generate avif and webp images, the following settings work :

luroConnect WatchImage Configuration

How does it work?

A watch is setup for the folder mentioned and when a new image is saved in any folder or subfolder, the image conversion will automatically be launched and the formats specified will be generated, with the same base name and corresponding extension.

For example If we are watching

pub/catalog/product/cache folder and an image with the name 3af406316384c53a5f29d3772fa411e5_2.jpg is saved the following images will be generated

pub/catalog/product/cache/3/a/3af406316384c53a5f29d3772fa411e5_2.webp

pub/catalog/product/cache/3/a/3af406316384c53a5f29d3772fa411e5_2.avif

The developer can generate the picture element with srcset for mime types image/webp and image/avif with the same base file name but with the corresponding file extensions.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like Image Optimization, ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call

Use of luroConnect WatchImage Service

This is best suited when the HTML code uses <picture> element with fine tuned image sizes (and mime types).

Information about <picture> element is found here.

HTML supports the picture element which allows a developer to define various image sizes and formats and specify conditions – much like css media query – for the browser to decide what is the closest match.

Briefly, using <picture> as the parent tag and <srcset> to specify conditions and alternatives.

<picture>
<source srcset="catalog/product/cache/xxx_for_600_width/m/y/myimage.jpg" media="(min-width: 600px)" />
<source srcset=" catalog/product/cache/xxx_for_600_width/m/y/myimage.webp" type="image/webp" media="(min-width: 600px)" />
<img src="myimage.jpg"/>
</picture>

January 23, 2024 By Core Web Vitals, Image, Magento, Site Speed 0 Comments Read more...

luroConnect Magento CI/CD adds meta information to builds

luroConnect CI/CD

luroConnect managed hosting platform includes CI/CD from any git source.

luroConnect CI/CD is an opinionated, platform specific CI/CD. It currently supports Magento (luma, Hyva, ScandiPWA themes), PWAStudio, Angular, nodejs applications. Nodejs builds include yarn and npm based installation.

Builds are done in docker with environment variables. (Note : Our Magento CI/CD also builds in docker without a database).

Builds once made, are ready to be deployed.

Advantage of opinionated CI/CD

  • Built for the target application stack with options.
  • Improved over time in a platform specific way
  • Magento CI/CD builds support features like magepack bundling, css and js minification, luma, Hyva and ScandiPWA themes.
  • Magento CI/CD support multi website and multi store builds
  • PWAStudio supports multi website and mutli store builds with different default currencies.
  • Is built using resources on your AWS account and created to deploy on your AWS account.

Meta Data information

Builds are identified with git commnit-id (shortened). Meta information includes the branch, the build number (generated from git as a serial count of commits in the branch upto this commit-id), and the git comment for this release.

luroConnect dashboard now displays meta data with the current deployed version.

When deciding which version to deploy, meta data information is displayed as well.

Build counts are useful as they are serial numbers – a larger number indicates a later commit serially. Commit comments are useful to know the PR or merge comment used so one is sure of the deployment.

luroConnect build and deploy dashboard

Magento CI/CD metadata

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call
August 18, 2023 By CI CD, hosting, Magento, PWA 0 Comments Read more...

Using AWS Autoscale “warm pools” to reduce costs

AWS Autoscale added a new feature “Warm Pool”.  Let us explore this feature and see how luroConnect uses this to reduce hosting costs.

The autoscale latency problem

Usually, AWS Autoscale will launch a new server with the given AMI image based on the launch configuration or launch template configured. Launching a new server takes about 4 minutes or more. So let us say a scale-out event is configured for launching a server when the CPU across all autoscale instances exceeds 70% for 1 minute. Now, let us say a sale promotion on facebook causes a surge in traffic causes this event to trigger. It takes AWS 4+ minutes to respond and add a new server. If during this 4 minute period, the surge goes past 70% and say reaches 90-100%, it is likely that visitors will see a slowdown or even errors. The 4+ minute period is called the autoscale latency and in designing the scale-out and scale-in parameters, it plays a crucial role.

For a website that sees frequent surge in traffic in short spurts, one would be prompted to use a lower threshold for a scale-out event. A lower threshold will result in frequent triggering of scale-out events.

At the same time the scale-in threshold will also have to be reduced to ensure enough spread between scale-out and scale-in events. A lower spread will result in an unhealthy sequence of a scale-out event adding a resource for it to be immediately removed.

Autoscale designers then tend to add higher number of minimum instances, possibly of larger sizes. That reduces the effectiveness of autoscale – and increases AWS costs.

Lowering the autoscale latency results in a better autoscale system. As the latency reduces, the need for larger number of minimum instances or larger size instances reduces. This results in savings in the AWS bill.

Introducing the warm pool

AWS now introduces the concept of a warm pool. The costs saving of a warm pools come from AWS policy for not charging for instances in stopped state – except for the disks. A warm pool is a set of autoscale instances that are launched but kept in stopped state. When a scale-out event happens, the latency Is now reduced to the boot time of an instance and any initialization needed – we measured adding 3 instances took about 35 seconds to start serving traffic for Magento.

A scale-in policy simply stops the selected instance and add it back to the warm pool.

Warm Pool For Autoscale

How to use a warm pool?

If you are using launch template for your autoscale, creating a warm pool is easy and documented here. If using lifecycle events, newer events have been introudced.

If using a launch configuration, we suggest upgrading to a launch template before using a warm pool. While upgrading to a launch template is easy, it is advisable to read about launch templates as they are a different and a larger concept.

Changing your instance image when in a warm pool

AWS has support for “instance refresh” – a term used by AWS to indicate an update in AMI image for all running and warm pool instances in a single command. However, this update has a crucial flaw – it can keep your website inaccessible for a short time. This is due to AWS terminating an instance before adding one. If an image has to be updated – such as a new code deploy – a custom strategy has to be deployed to ensure the website does not go down.

luroConnect support for warm pool

luroConnect now supports warm pools across all its autoscale plans, with a scripted image update policy that ensures 0 downtime during image change as well as a code deploy strategy that ensures 0 downtime on code deploy.

Issues with AWS Reference architecture and tools for a Magento application

At luroConnect we implemented our autoscaling system after addressing flaws in many implementations we had seen.

As AWS autoscale by default is integrated into AWS load balancers – ELB or ALB. Using AWS reference implementation will put the code in a autoscale instance with nginx or apache with php and the code. Traffic can be routed through the ELB/ALB which will handle SSL and route the traffic to each autoscale instance.

When code has to be updated, a new AMI will be created and AWS instance refresh can be run to update the instances.

You could use AWS CodeDeploy as described here but you need to set it up to make sure Magento setup upgrade can be run when required.

Problems with autoscale implementations for Magento

  1. Issues configuring FPC (Full Page Cache) with this configuration : If varnish is configured on all autoscale instances (as we have seen many implementations do), each server will warm caches on its own. Clearing pages from cache will also be difficult. Using redis as a FPC increases per page latency for cached pages.
  2. Media and var folders are needed to be shared across all servers. NFS is typically used to share. However, the configuration of each autoscale instance has to be such that it can discover and mount the folders from the NFS server.
  3. When a code change has to be deployed, it is not clear how it can be done without causing a downtime of the website. Using AWS Code Deploy requires a complex setup to ensure setup upgrade is run before one of the 0 downtime strategies can be used.
  4. When a new server is launched, conditions to check the health of the website are not easy to write. This results in a few error responses before the server is ready to serve traffic.
  5. It is difficult to use a AWS ALB to route traffic for specific purposes – for example, routing traffic to a wordpress server for /blog urls.

luroConnect Autoscale on AWS : Smooth setup and running.

luroConnect Autoscale solves these problems.

luroConnect lets AWS monitor instances and decide when to add or remove (scale out or scale in) instances. luroConnect autoscale for AWS adds cloudwatch events and lifecycle management generated by AWS Autoscale to ensure a very smooth Autoscaling operation. luroConnect uses nginx as a load balancer and does not require a ALB/ELB to operate. luroConnect Autoscale supports AWS Autoscale with warm instances and has a mechanism to update the AMI when needed without any downtime.

  1. Using nginx as a load balancer allows high flexibility in deciding which urls go to varnish for full page cache and which should be directly served by php. varnish as a full page cache gives the maximum impact of full page caching.
  2. A nfs server holds shareable content of magento – specific media and var folders for example. Using NIS, autofs and NFS, each new app server is able to discover the NFS share.
  3. When a code change has to be deployed, php code using nfs is shared to each app server. A php reload and opcache configuration will ensure the new code is kept in the php opcache memory for all future operations. A php file from NFS share is loaded only once.
  4. Before a server is added to the nginx load balancer, extensive checks are done to ensure the new autoscale instance is ready to take traffic, including warming the opcache.
  5. nginx as a load balancer brings in a lot of flexibilty in routing traffic such as a /blog to a wordpress website, custom rewrites, etc.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call
June 10, 2023 By devops, Horizontal Scaling, hosting, Magento, Nginx 0 Comments Read more...

Do you know what size server you want for your eCommerce site?

Leaving the toughest question unanswered

When signing up for your Magento hosting, the first question you see asked, before you place an order, is what size server you want. It has become so ubiquitous, that everyone just expects to answer it looking at the cost.

But this is much like Mathematics books leaving tough problems as exercise to the readers!

It should not be that way! The size and architecture of the server you need depends on many factors.

Factors to consider

  1. The traffic and pattern. We routinely ask for 2 google analytics graphs – one for a typical day and one for a high sale day. This drives the architecture and server size.
  2. Your hosting stack – are you vanilla magento? or do you use headless / PWA? or use some software for image optimisation on your server?
  3. If the live site is already hosted, current CPU and memory usage.
  4. The size of the magento database.

luroConnect always starts an engagement with a server sizing sheet that is filled on behalf of the merchant. This allows us to propose a hosting plan on the customers cloud account and an appopriate luroConnect support plan.

Take the guesswork out of server sizing with horizontal scaling

A classic 3-tier architecture.

  • The web layer (WAF, apache / nginx /varnish, cron, rabbitmq),
  • the application layer (php, nodejs) and
  • the database layer (mysql, elasticsearch, redis).

Horizontal Scaling :

  • app servers can scale independently – indeed they can be autoscaled.
  • Low traffic websites can fold either the app or the db layers or both into the web layer
  • The db layer can be extended to have master slave
  • A proxy layer can load balance read traffic between master and slave, giving scalability at the database level

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call
March 28, 2023 By hosting, Magento, Site Speed 0 Comments Read more...

Improving PDAStudio performance : Cache more

PWAStudio makes many “storeConfig” calls that – depending on the website – be made cacheable in varnish as they are unlikely to change. This will reduce cookie blocking – or reduce load on php if the authorization is used.

What is the session blocking problem?

When Magento (php) processes a request, the cookie passed is mapped to a session in Magento. Sessions are stored either in var/session or on production systems, in redis.

Magento will lock the session until the php code processes this request. At the end of the request the session is unlocked.

When the session is locked, any hit with the same cookie, is kept in wait state, waiting for the session lock to be released. If there are more than one hits, all hits wait. These are not kept in a ordered queue – instead they each check the state of the session every x seconds and the first one breaking in gets the session lock.

Session lock is important to process hits that update the session – login, cart operations, checkout operations can use sessions. But, the problem is that, not all hits need this lock, and Magento gives no way for a developer to hint that the hit being processed does not need a lock.

The problem with ajax calls – especially PWAStudio – is even more severe. To display a page, many calls for each component on the page are made. If all of these are to be processed by magento, they will all be queued and a single slow query can bring down the site performance.

A PWAStudio Example

As can be seen in the screen below, a page refresh of a PWAStudio website hosted on Magento Commerce Cloud, The arrows highlight the hits that are marked by Magento as non cacheable.

Analyzing the content returned by the getStoreConfig hit.
https://www.soch.com/graphql?query=query+GetStoreConfigForCarouselEE{storeConfig{store_code+product_url_suffix+magento_wishlist_general_is_enabled+enable_multiple_wishlists+__typename}}&operationName=GetStoreConfigForCarouselEE&variables={}

It is clear that these values are unlikely to be changed and can be cached by TTL alone – with a potential ability to clear varnish with a URL / regular expression to help clear the url if an occasion arises.

Improve caching

The hits marked in Blue on the right have been analyzed by luroConnect and we can suggest to cache in varnish on TTL of 1 day. Note : Varnish cache is already hashed with the store key. This is important as some of the data is store specific.

This requires a vcl change.

Yet recommending Magento Commerce Cloud for hosting?

Think again! Analysis like this is only available from luroConnect. Moreover, we will implement the vcl changes.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call
March 3, 2023 By caches, Magento, PWA, Site Speed, varnish 0 Comments Read more...

Meet Magento India 2023 – Speaker Talk – Improve PWA Performance

On Feb 3, 2023, our CEO Pradip Shah presented a talk titled “Improving Headless / PWA Performance” at Meet Magento India in Mumbai, India.

The premise of the talk was the observation that the Headless / PWA eCommerce world promised lightning fast websites but many have ended up with slow performance, especially as measured by Google’s metrics.

This talk discussed issues and solutions that have been implemented in real eCommerce websites to increase page speed.

There are 2 versions of this for you to download

  1. The pdf version (5MB)
  2. The powerpoint version (23MB) – it has a few videos, and hence recommended to download.

Summary

Section 1 : Legacy HTML page technology is not stagnant – with Hyva and on page optimization from technologies like Nitrogen, page speeed scores of Magento traditional HTML page has improved. However, many traditional websites suffer from slower response speeds with uncached pages. Example : www.kalkifashion.com is a Hyva theme with Nigrogen front end caching and optimizaion.

Section 2 : PWA is not magic – the promise of “lightning fast” website needs work. Basic page speed principles apply and so do some new ones.

Section 3 : It is possible to make websites using Magento REST and Graphql. Each one has different challenges. REST by default does not cache and Graphql uses sessions.

Section 4 : When using graphql, avoid POST graphql for queries – use them only for mutations.

Section 5 : Study your application and you can add caching to Graphql’s that Magento may mark as not caceheable.

Section 6 : Custom graphql queries result in database queries that may need optimization. A step by step method to get the underlying mysql query generated is a useful process. We used that to achieve great results. www.mysoresareeudyog.com has a custom default filter for a listing page – latest.

Section 7 : Magento 2.4.5 allows method to prevent session locking using authentication. But by default, authenticated graphql calls are not cached – an approch similar to one presented earlier will be required to ensure optimal caching is achieved.

Section 8 : Go beyond. Add logic in frontend and cache GraphQL results in the browser. This is the ultimate power of headless / PWA. It may not help with page scores that are measured without browser cache, but user experience will improve.

Section 9 : Bulid can improve page loads. Appropriate sized chunks and using hashes instead of version. Another improvement build can give is use of cookie free domains. A static domain for example for js and css bundles allows the main domain to not go through a caching layer, improving page load speed.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call
March 3, 2023 By Magento, page speed, performance, PWA, Site Speed 0 Comments Read more...

Should I move to shopify from Magento?

The platform battle of Magento vs Shopify rages on. Increasingly merchants are moving to Shopify from Magento due to easy of operations and website management.

For many, where the decision was a no-brainer, the move has already been done.
For others, this question rages on, as shopfiy adds more features from itself and its partners in an easy to manage.

This article is written in 2022 for the “others”.

The “others” I refer to have some of these characteristics

  • A working eCommerce business – one that makes money, increasing the risk of a move
  • Migrated to Magento 2 and hence have invested in the technology
  • Customizations that differentiate the store and would be difficult to achieve in Shopify. These may include custom attributes, use of custom options, personalization technology, etc.
  • Headless frontend
  • Worried about data ownership

What are the pain points in Magento that makes merchants look at shopify

  • Managing the Agency or Magento Agencies are expensive!
  • Website speed scores or Magento website speed scores can be bad.
  • Everything is so hard in Magento compared to Shopify

What options you have to ease the pain and stay on Magento?

  • Managing the Agency.In my conversation with merchants unhappy with Magento, the main reason to consider a move is the performance of the Agency – i.e. they have an agency problem, not a platform problem.
    • Agencies have powerful marketing presentations that appeal to the heart of an eCommerce merchant. But, a merchant should pay attention to the compatibility between of the agency to how the business operates before choosing one. Infact, it may be even desirable to manage multiple agency relationships.Some factors to consider.
    • If you have an in-house project manager, you need developers – an strong in development. One of our customers has 2 agencies – a cheaper local and a more established one. Depending on the complexity of the project, they decide which agency will be best. They have their own project manager.
    • Some agencies have a delivery manager that acts as a single point of contact. They work with you on your requirement, discuss with the developers and are responsible for the delivery. Ensure they communicate with all teams – including business and SEO.Some agencies offer a full eCommerce service – including SEO to development. These are very suitable for growing eCommerce arm of a omni channel merchant. Such agencies can work on outcome basis – with a profit share for example.
    • A point of conflict between merchants and agencies is the business model – Agencies like an hourly billing model as it closely matches their biggest cost – the manpower used to execute the project. Many merchants may think of a fixed cost model best suits them – with outcomes well defined. Software on the other hand is very difficult to estimate, especially in an agile development world. Merchants wanting a fixed cost model should be more acceptable to cost escalations due to change requests.
    • Use an expert to interface with an agency. If you have a complex project and do not have an inhouse eCommerce manager, it will be a good idea to hire an consultant to interface with the agency. The consultant can act as a project manager communicating the business needs to developers and ensuring the delivery meets the quality goals.
  • The user interface. From a competitive perspective, UI is the most crucial part that can help you differentiate from your competitors. A more personalized website or an emersive experience will attract more visitors to stay and purchase.But, magento websites are slow on page speed scores, especially those developed on the original Magento 2 theming, which is archaic by todays UI technology standards.The Magento UI stack is evolving. Apart from the many headless options, Hyvä Themes has completely rewritten the theme to be fast and modern.Infact, in my opinion, UI should be the main reason to stay on Magento – the ability to give the best user experience. Rather than have a single page speed goal for the website, have a more user experience view. Even google suggests to have targets for faster website and work towards achieving them.
  • Ease of Shopify vs Magento. A big complaint against Magento is its bloat – too many features, not many, easy to use. A customer recently made this comment when they were trying to enable the watermark in Magento. Turns out, while the watermark feature is shown in Magento’s admin dashboard, enabling it will immediately remove all images – until the developer had accounted for this in their development. The feature bloat meant the developer did not really understand why it would not work. While this feature just works out of the box in shopify.

Magento has a community

Magento has over the years grown to have a very vibrant community. Some estimates put it at over 300,000 strong. Mainly tech professionals, it also consists of many areas of eCommerce business.

For some merchants, it is important to know they (or their online business) is not beholden to a single corporation. A Shopify network for example, though very cohesive, is orchestrated by Shopify. Shopfiy does whet these vendors, but there is no assurance of quality given by Shopify on their behalf. Moreover, Shopfiy can decide to compete with one of the features with any partner in that network, making your favourite plugin obsolete.

Magento on the other hand has independent plugin developers, some of who have made a name for themselves. The agency and merchant have the task of ensuring every plugin they use is good in terms of quality and functionality.

Migration to shopify may not be all roses

On the other hand, shopify’s ease of administration can get in the way of large changes. A process has to be used including using staging site for UAT, git and tools like ThemeKit, Beanstalk and/or the shopify github app. Which brings back the agency and project management question.

Getting feature parity with a Magento website may also not be easy. One merchant moved to Shopify and lost some upsell opportunities as they had used custom options in Magento. They commissioned some custom development and realised that the agency would launch a server and store some data there. This added to a vendor lockin (their agency) they did not want.

luroConnect easing the pain of Magento and Headless/PWA

luroConnect is a managed hosting platform for eCommerce – including Magento and Headless eCommerce.

We believe in serving 2 stake holders in an eCommerce website – the developer / agency and the merchant. We have tools and features to help both. To remove the pain of taking developer code to production. luroConnect supports dev, staging and production environments, with an inbuilt CI/CD for all platforms we support, with low downtime deploys.

The luroConnect stack deploys on the cloud of customers’ choice. The production site is backed by a performance dashboard, configurable alerts and 24×7 support.

Our extensive agency network helps us match a merchant to an agency.

Considering moving from Magento to Shopify? Talk to us. We may just give you a reason to stay back!

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.
Click here to schedule the call
September 24, 2022 By Magento, PWA, SEO, Site Speed 0 Comments Read more...

Magento and graphql : A slowdown saga

PWA, Magento and graphql : A slowdown saga

Note : I write this article as Magento 2.4.5 is released and fixes some major performance issues that have resulted in the dream of headless and PWA go sour for many early adopters of technologies like PWA Studio and Vue Storefront 2.

PWAs and headless came with the promise of fast navigation and have had a rough ride at best, especially with Magento backends. Early PWAs like Vue Storefront 1 were implemented using Magento Rest API with a api middleware in nodejs that used elastissearch to store cached version of magento catalog for faster access. Magento implemented Graphql to avoid the need for the rest api. Use of direct rest api is ill-advised without a middleware as it exposes a Magento admin token. A middleware would sanitize the request possibly by checking a cookie and also limit the range of queries run. Graphql prevented the use of admin tokens and instead relied on frontend cookies for authentication.Magento’s REST API also results in large datasets. Graphql has a powerful, standardized query syntax to theoretically present a path breaking solution to build Magento headless around. However, early implementations did not fully cover the entire magento data model – Magento even advised the use of REST api when graphql was not sufficient. And when real world websites with large builds and catalogs were deployed, performance issues emerged.

Graphql are ajax calls in Magento that use sessions. In a typical hit processed by Magento, a session is opened, the hit is processed and then the session is closed. (A hit here is either a frontend hit or an ajax call).

Sessions and race conditions

Graphql are ajax calls in Magento that use sessions. In a typical hit processed by Magento, a session is opened, the hit is processed and then the session is closed. A hit is either a frontend hit or an ajax call.

Magento can store information in sessions – cart data for example. In order to prevent 2 hits from writing contents to the session simultaneously – called a race condition, sessions are locked for the duration of the processing a hit. (Take for example a checkout flow and let us assume that the address and payment info is written to session. You would want to ensure both get written. If both “raced”, only one piece of information would be written). Magento identifies an internal session by setting a cookie in the visitors browser. Each hit carries the cookie allowing magento to identify the session. Since a session is unqiue for a visitor, so Magento can process process only one hit per visitor at any given time.

With ajax in general and graphql in particular, many simultaneous hits may go to the server. Magento, through the session locking mechanism, queues them up. The hits are not guaranteed to be processed in the order they were received or any order in particular.

A typical session content in redis looks like

hgetall sess_eba681783f7000bc668cc45005aeaca4
1) "lock"
2) "0"
3) "writes"
4) "5"
5) "data"
6) "_session_validator_data|a:4:{s:11:\"remote_addr\";s:14:\"116.58.200.181\";s:8:\"http_via\";s:0:\"\";s:20:\"http_x_forwarded_for\";s:0:\"\";s:15:\"http_user_agent\";s:117:\"Mozilla/5.0 (Linux; Android 10; RMX2030) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Mobile Safari/537.36\";}_session_hosts|a:1:{s:10:\"host.com\";b:1;}default|a:0:{}customer_base|a:2:{s:11:\"customer_id\";N;s:17:\"customer_group_id\";N;}checkout|a:0:{}"
7) "wait"
8) "0"
9) "pid"
10) "10.0.2.197.host.com|3735"
11) "req"
12) "POST host.com/graphql"
Most of these entries are used internally by the redis session module. Entry 6 is the main session, stored in php serialized format.

If each hit takes only a few milliseconds and the number of ajax calls are not high, the overall response time may not perceptibly slow the site.

But, if even one ajax hit takes time, the page load can slow down. If a customer opens multiple tabs or refreshes the page, the slowdown may be more severe, sometimes leading to errors.

Here is an example. We are analyzing server logs based on IP. Lines 1 and 2 indicate the same link in google was clicked twice – possibly in 2 tabs on the browser. As the page loads we see graphql’s being loaded. The last 2 items are an indication of the problem.

The 2nd last hit is the slow hit – it took 10 seconds to return a result as the item was not in cache. The problem in the last hit was that it was waiting for the 2nd last hit to be processed, although in all probability it was in cache. If the number of items in this queue grows above 6 – for example if a visitor opens tabs and accesses urls not in varnish, a 500 error may be returned, possibly resulting in the UI showing a block not fully loaded at all. A refresh typically seems to solve the problem.

Server can run out of php resources

When too many graphql calls wait, server side can run short of php resources. This time you can see 504 and 502 errors. Adding more php resources for the same server risks running out of RAM. Adding more servers does help with the problem, but that is a waste of resources.

Graphql and caching issues

Graphql in Magento needs many caches to perform well. However, upon cache clear the “galloping horses” problem can bring down a busy website.

Upon cache clear the first access to graphql results in the schema being read. The performance for this action has improved substantially since Magento 2.3, but we yet highlight it as a concern. A cache warming of this is preferred.

Queries like filter in a category page can be slow and hence caching and warming some queries is always a good idea.

Enabling varnish cache is also important – varnish does not use sessions.

Recommended way to use Magento graphql

Graphql needs authorization – it needs to know what customer is asking for the information. It needs it so that the results include say for example the correct tiered pricing for this customer. While sessions include this information, they also store state and hence need the protection of a lock.

Magento supports JWT customer tokens for graphql.
https://devdocs.magento.com/guides/v2.4/graphql/mutations/generate-customer-token.html

Use the generateCustomerToken mutation to create a new customer token.
Example :

mutation {
  generateCustomerToken(
    email: "bobloblaw@example.com"
    password: "b0bl0bl@w"
  )
}

Response

{
  "data": {
    "generateCustomerToken": {
      "token": "ar4116zozoagxty1xjn4lj13kim36r6x"
    }
  }
}

Magento suggests NOT enabling cookie and JWT token authentication at the same time to reduce the chances of encountering problems caused by the differences between the two authorization methods. However, disabling cookie was added a feature only in Magento 2.4.5

However, Magento 2.4.5 varnish sample configuration disables varnish caching for logged in user. This can reduce performance.

Conclusion

Magento performance continues to be a troublesome aspect of the eCommerce platform. At luroConnect we look at performance in great depths.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and many scaling options to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for eCommerce – specifically Magento and various PWA / Headless technologies.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimize hosting costs
Click here to schedule the call
September 12, 2022 By Magento, performance, PWA, Site Speed, varnish 0 Comments Read more...

Sansec reports new Magento 1 hack

Over the weekend of Sept 11, 2020, Sansec reported a web skimming attack on Magento 1 stores. It was the largest single day automated attach recorded by Sansec.

What are skimming attacks?

“Skimming” attacks are malicious code added to your website so when a site visitor is entering any personal information including credit card, the content is “skimmed” and sent to the attacker.

The website looks completely normal.

In September 2018, British Airways revealed that 380,000 passenger information had been skimmed from the website. The modus operandi for this attack was access to the code (possibly the version control of a 3rd party javascript module used on the website). The attack went undetected for months.

Some attacks are also called “Magecart” attacks.

In Magento we see 2 popular ways

  • Break the admin password and upload content to “Miscellaneous Header” or “Miscellaneous Footer” sections.
  • Upload a php code file which in turn loads the real malicious code in javascript in the page – either directly into the page or modify a known javascript file.

The current attack is of the second variety.

Stores on luroConnect were not attacked!

The attack as described by Sansec in the article used the Magento connect to bypass Magento admin and upload malicious code in javascript files to facilitate skimming of credit card information.

luroConnect has many rules that helped prevent this attack from affecting any of our Magento 1 websites.

Rule 1 : “/downloader” URL  is not accessible on any live or staging website. We expect code to be deployed through git and expect the developer to use a manual process to install modules. We disallow magento connect based installation in any of our managed websites.

Rule 2 : Our web directory owner and hosting users are different. Hosting user is the user php code runs as. Moreover, /skin folder is not writable by the hosting user.

Rule 3 : We use a static minifier and deploy the code to a folder skin.min which is not in git. The /skin folder itself is never used.

Rule 4 : Staging and dev environments are protected using a HTTP Basic Authentication. Automated attack vectors would need to add a password guesser before they can reach the staging URLs. This is assuming a developer would have relaxed permissions in the dev environment.

Rule 5 : Our platform bars ssh access to the hosting user. This prevents any accidental change in permissions being permanent. Even in the rare case ssh access is given (for debugging purposes), upon relinquishing the access, we sanitize the environment with default permissions.

How to protect your store?

One of the best ways is to sign up for Sansec’s security scanner eComscan

luroConnect is a very secure platform for Magento hosting. We call it layered security – from a secure file system and strict folder permissions, to an inbuilt WAF with configurable rules to partnering with Sansec for security scans.

We host you on your cloud or physical hardware using our stack. Learn more about our plans here.

September 15, 2020 By hosting, Magento, Nginx, Security 0 Comments Read more...

Supercharge your Magento with a Varnish cluster

What is a varnish cluster?

A varnish cluster is a set of varnish nodes, each in a different geographical location, in front of the same Magento backend.

As shown a Magento hosted in the US East region can serve varnish nodes from across the world. Access to the website from each region is directed towards the nearest varnish, benefiting from lower latency and faster page loads.

If you serve customers in different regions – internationally or across the USA, your store can benefit from a varnish cluster.

Varnish and Magento performance

Magento 2 was architected to work with Varnish for improved performance. A typical webpage – category listing or product detail page – when returned from a cached varnish page (called a HIT in varnish) typically has a TTFB of a few milliseconds. An uncached page (also called a MISS in varnish) results in TTFB of a few seconds. Optimizing a MISS is very crucial, but we will not cover that in this article. We believe varnish is one portion of optimizing your website.

What is Network Latency

Latency is the time it takes for a network packet to go from your computer to the server with a request and come back with a response – assuming the response from the server was available immediately. When a page scores a HIT in varnish, the response is almost immediate from the server. Any  Time To First Byte (TTFB) recorded on the browser can be attributed to network latency.

If you market your website to many regions around the world, and you host varnish in a single location, your visitors may be faced with higher latency. When latency of access is in a few hundred milliseconds, it becomes the bottleneck and needs optimization.

The below information from https://wondernetwork.com/pings/ gives an idea of expected latencies. To read the table use the row header as source and column header as destination. So, a ping latency from Los Angeles to Mumbai would be 267 msec.

What is the cause of latency?

Core latency is a function of distance – even light will take ~40msec to travel 13,000 km – the distance from New York to Mumbai. A network packet travels through wires at that speed, and network wires are much longer than a straight line. Moreover, a response packet has to travel all the way back.

Latency is also caused as traffic goes through network equipment. The number of switches / servers a packet has to go through depends on the network and service providers – yours as well as at the server you are connecting to.

A varnish cluster architecture

Using a GeoIP based DNS service such as AWS Route53 a users request for a domain is redirected to one of several IPs. The Magento backend is in the “default” region – where maximum traffic is expected. A varnish in the regions desired – shown US East and Europe below.

Geo-IP based DNS

A geoip based dns router such as AWS route53 can help direct traffic to the nearest varnish node based on the guessing the country the IP requesting name resolution is coming from. So users on a browser say in Australia would be directed to be served from the varnish node in Australia and one from the west states of California would be directed to the varnish there.

Since IP to region or country can never be so accurate, it should be possible for all varnish nodes to serve customers from any region. Specifically, a language or currency switcher should be available on the website.

Magento 2 supports multiple varnish nodes out of the box

An important feature in using a cache in production is its need to automatically and quickly clear contents on demand from the user and application. For example, when a product goes out of stock, the corresponding page should display the out of stock label. Magento uses a tag-based system to flush appropriate content from the cache. Magento allows setting up multiple varnish hosts and a tag-based cache clear is sent to all the hosts.

bin/magento setup:config:set --http-cache-hosts=<varnish internal ip>:6081,<varnish internal ip 2>:6081

Refer : https://devdocs.magento.com/guides/v2.4/config-guide/varnish/use-multiple-varnish-cache.html

Challenges in a varnish cluster

A varnish cluster is more complex to manage

Ensure the varnish vcl files, front end security configuration (WAF, rate limiting, etc) is managed and kept in sync on all edge nodes.

Managing includes monitoring to ensure none of the servers go down. Now, your site can be down in a specific region for example. Typical monitoring tools such as Pingdom would not work. A purposeful monitoring solution is needed.

A varnish cluster costs for additional servers

Since these would be frontend servers, the amount of RAM and their network speed requirement would depend on what traffic they get.

Number of varnish nodes

Increasing the number of nodes in a varnish cluster does not always help in improving site speed. That is because each varnish node has a different hit ratio. A lower hit ratio leads to more users getting the latency and performance penalties combined – due to a varnish MISS. Traffic pattern and latency have to be taken into account to decide on how many nodes to use in a cluster.

Difficult to warm the cache

Given the distributed nature of the cache, warming each cache independently takes more resources on the server side as well as some changes to the way a cache warmer works.

luroConnect : A modern cloud hosting platform

Schedule a call of a free evaluation and demo!

  • Is horizontal scaling manually or with autoscale right for you?
  • Evaluate if a varnish cluster will help your website performance
  • Show managed dev, staging and production environments
  • How we measure application performance every minute with alerts for a slowdown
  • Can your hosting costs be optimized?
  • How improved hosting can lead to better ROI!

On Magento Cloud? We have special offers if you switch your enterprise license to luroConnect managed AWS cloud.

Click here to schedule the call

Magento Cloud and Fastly

It is important to discuss the Magento 2 cloud decision to use Fastly as a frontend.

 

Magento 2 cloud pro version architecture is given below. (Reference : https://devdocs.magento.com/cloud/architecture/pro-architecture.html)

The architecture uses Fastly as a Full Page Cache. Varnish is not installed on the Instances in AWS.

Fastly is a CDN that uses varnish. With the Magento 2 Cloud integration, a custom plugin is used on Magento along with a custom vcl file that runs on Fastly.

Fastly has many “POP” locations. As per fastly documentation, there are 20 POP locations in North America, https://www.fastly.com/network-map

mostly in the USA. Each has its own varnish cache. When a page is not in a POP, it fetches content from the origin. A single page may have to be rendered 20 times for each POP location in North America.

Drawbacks of this architecture from a varnish cluster perspective

  • More POPs do not lead to better experience as a higher percentage of MISS on varnish results in a worse experience for more users.
  • As POPs increase the load on Magento infrastructure increases.
  • It is not possible to use a cache warmer to warm the Fastly cache.

What next?

An ideal situation would be a layered varnish configuration – each “satellite” varnish node serving a local user, caching a subset of the “main” varnish node, reducing the penalty of a varnish MISS.

Share your thoughts here or on social media.

August 25, 2020 By caches, hosting, Magento, Nginx, performance, Site Speed, varnish 0 Comments Read more...