varnish

Improving PDAStudio performance : Cache more

PWAStudio makes many "storeConfig" calls that - depending on the website - be made cacheable in varnish as they are unlikely to change. This will reduce cookie blocking - or reduce load on php if the authorization is used.

What is the session blocking problem?

When Magento (php) processes a request, the cookie passed is mapped to a session in Magento. Sessions are stored either in var/session or on production systems, in redis.

Magento will lock the session until the php code processes this request. At the end of the request the session is unlocked.

When the session is locked, any hit with the same cookie, is kept in wait state, waiting for the session lock to be released. If there are more than one hits, all hits wait. These are not kept in a ordered queue - instead they each check the state of the session every x seconds and the first one breaking in gets the session lock.

Session lock is important to process hits that update the session - login, cart operations, checkout operations can use sessions. But, the problem is that, not all hits need this lock, and Magento gives no way for a developer to hint that the hit being processed does not need a lock.

The problem with ajax calls - especially PWAStudio - is even more severe. To display a page, many calls for each component on the page are made. If all of these are to be processed by magento, they will all be queued and a single slow query can bring down the site performance.

A PWAStudio Example

As can be seen in the screen below, a page refresh of a PWAStudio website hosted on Magento Commerce Cloud, The arrows highlight the hits that are marked by Magento as non cacheable.

Analyzing the content returned by the getStoreConfig hit.
https://www.soch.com/graphql?query=query+GetStoreConfigForCarouselEE{storeConfig{store_code+product_url_suffix+magento_wishlist_general_is_enabled+enable_multiple_wishlists+__typename}}&operationName=GetStoreConfigForCarouselEE&variables={}

It is clear that these values are unlikely to be changed and can be cached by TTL alone - with a potential ability to clear varnish with a URL / regular expression to help clear the url if an occasion arises.

Improve caching

The hits marked in Blue on the right have been analyzed by luroConnect and we can suggest to cache in varnish on TTL of 1 day. Note : Varnish cache is already hashed with the store key. This is important as some of the data is store specific.

This requires a vcl change.

Yet recommending Magento Commerce Cloud for hosting?

Think again! Analysis like this is only available from luroConnect. Moreover, we will implement the vcl changes.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs

Magento and graphql : A slowdown saga

PWA, Magento and graphql : A slowdown saga

Note : I write this article as Magento 2.4.5 is released and fixes some major performance issues that have resulted in the dream of headless and PWA go sour for many early adopters of technologies like PWA Studio and Vue Storefront 2.

PWAs and headless came with the promise of fast navigation and have had a rough ride at best, especially with Magento backends. Early PWAs like Vue Storefront 1 were implemented using Magento Rest API with a api middleware in nodejs that used elastissearch to store cached version of magento catalog for faster access. Magento implemented Graphql to avoid the need for the rest api. Use of direct rest api is ill-advised without a middleware as it exposes a Magento admin token. A middleware would sanitize the request possibly by checking a cookie and also limit the range of queries run. Graphql prevented the use of admin tokens and instead relied on frontend cookies for authentication.Magento's REST API also results in large datasets. Graphql has a powerful, standardized query syntax to theoretically present a path breaking solution to build Magento headless around. However, early implementations did not fully cover the entire magento data model – Magento even advised the use of REST api when graphql was not sufficient. And when real world websites with large builds and catalogs were deployed, performance issues emerged.

Graphql are ajax calls in Magento that use sessions. In a typical hit processed by Magento, a session is opened, the hit is processed and then the session is closed. (A hit here is either a frontend hit or an ajax call).

Sessions and race conditions

Graphql are ajax calls in Magento that use sessions. In a typical hit processed by Magento, a session is opened, the hit is processed and then the session is closed. A hit is either a frontend hit or an ajax call.

Magento can store information in sessions – cart data for example. In order to prevent 2 hits from writing contents to the session simultaneously – called a race condition, sessions are locked for the duration of the processing a hit. (Take for example a checkout flow and let us assume that the address and payment info is written to session. You would want to ensure both get written. If both “raced”, only one piece of information would be written). Magento identifies an internal session by setting a cookie in the visitors browser. Each hit carries the cookie allowing magento to identify the session. Since a session is unqiue for a visitor, so Magento can process process only one hit per visitor at any given time.

With ajax in general and graphql in particular, many simultaneous hits may go to the server. Magento, through the session locking mechanism, queues them up. The hits are not guaranteed to be processed in the order they were received or any order in particular.

A typical session content in redis looks like

hgetall sess_eba681783f7000bc668cc45005aeaca4
1) "lock"
2) "0"
3) "writes"
4) "5"
5) "data"
6) "_session_validator_data|a:4:{s:11:\"remote_addr\";s:14:\"116.58.200.181\";s:8:\"http_via\";s:0:\"\";s:20:\"http_x_forwarded_for\";s:0:\"\";s:15:\"http_user_agent\";s:117:\"Mozilla/5.0 (Linux; Android 10; RMX2030) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Mobile Safari/537.36\";}_session_hosts|a:1:{s:10:\"host.com\";b:1;}default|a:0:{}customer_base|a:2:{s:11:\"customer_id\";N;s:17:\"customer_group_id\";N;}checkout|a:0:{}"
7) "wait"
8) "0"
9) "pid"
10) "10.0.2.197.host.com|3735"
11) "req"
12) "POST host.com/graphql"

Most of these entries are used internally by the redis session module. Entry 6 is the main session, stored in php serialized format.

If each hit takes only a few milliseconds and the number of ajax calls are not high, the overall response time may not perceptibly slow the site.

But, if even one ajax hit takes time, the page load can slow down. If a customer opens multiple tabs or refreshes the page, the slowdown may be more severe, sometimes leading to errors.

Here is an example. We are analyzing server logs based on IP. Lines 1 and 2 indicate the same link in google was clicked twice – possibly in 2 tabs on the browser. As the page loads we see graphql’s being loaded. The last 2 items are an indication of the problem.

The 2nd last hit is the slow hit – it took 10 seconds to return a result as the item was not in cache. The problem in the last hit was that it was waiting for the 2nd last hit to be processed, although in all probability it was in cache. If the number of items in this queue grows above 6 – for example if a visitor opens tabs and accesses urls not in varnish, a 500 error may be returned, possibly resulting in the UI showing a block not fully loaded at all. A refresh typically seems to solve the problem.

Server can run out of php resources

When too many graphql calls wait, server side can run short of php resources. This time you can see 504 and 502 errors. Adding more php resources for the same server risks running out of RAM. Adding more servers does help with the problem, but that is a waste of resources.

Graphql and caching issues

Graphql in Magento needs many caches to perform well. However, upon cache clear the “galloping horses” problem can bring down a busy website.

Upon cache clear the first access to graphql results in the schema being read. The performance for this action has improved substantially since Magento 2.3, but we yet highlight it as a concern. A cache warming of this is preferred.

Queries like filter in a category page can be slow and hence caching and warming some queries is always a good idea.

Enabling varnish cache is also important – varnish does not use sessions.

Recommended way to use Magento graphql

Graphql needs authorization – it needs to know what customer is asking for the information. It needs it so that the results include say for example the correct tiered pricing for this customer. While sessions include this information, they also store state and hence need the protection of a lock.

Magento supports JWT customer tokens for graphql.
https://devdocs.magento.com/guides/v2.4/graphql/mutations/generate-customer-token.html

Use the generateCustomerToken mutation to create a new customer token.
Example :

mutation {
  generateCustomerToken(
    email: "bobloblaw@example.com"
    password: "b0bl0bl@w"
  )
}

Response

{
  "data": {
    "generateCustomerToken": {
      "token": "ar4116zozoagxty1xjn4lj13kim36r6x"
    }
  }
}

Magento suggests NOT enabling cookie and JWT token authentication at the same time to reduce the chances of encountering problems caused by the differences between the two authorization methods. However, disabling cookie was added a feature only in Magento 2.4.5

However, Magento 2.4.5 varnish sample configuration disables varnish caching for logged in user. This can reduce performance.

Conclusion

Magento performance continues to be a troublesome aspect of the eCommerce platform. At luroConnect we look at performance in great depths.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and many scaling options to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for eCommerce - specifically Magento and various PWA / Headless technologies.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimize hosting costs

Varnish, Magento and frequent product updates

In response to Erwin Hofman’s M2 performance post, a reader commented

Varnish and Magento 2

Use of varnish with Magento has been a popular speed increase. So popular, that Magento decided to incorporate in M2.

This has spread endless debates though

  • Would it not have been better to improve the core code quality instead?
  • How often and which pages do we put in cache?
  • How to organize user personalization content? (Magento has answers in terms of ajax and ESI, but I am afraid the documentation is so spread around, many developers and plugin vendors are not sure).

The question raised by the reader is another point of debate – how can a Magento store, with frequent product updates be made ready for using varnish.

Let us see how Magento works with Varnish – or what it means to say M2 supports varnish.

(Refer this article for a detailed analysis.)

Magento cache has always (yup M1 also) supported tags in caches. A typical cache is a key->value pair, typically implemented as a hash table to lookup the key and return the value. In case of a full page cache, the key is made up of the the URL and parts of the HTTP header that forms the request. When the key is found in the hash table lookup operation, it is a hit and the content is returned. When the key is not found, the request is forwarded to magento to process it. When Magento processes it, the key and the content is stored. The content is the entire HTTP response, including HTTP response headers and HTTP body.

When Magento processes a request, it returns in a HTTP response header, a “hint” header called “X-Magento-Tags”. This header has information about what type of content the page holds.

For example, a category page will have tags that represent the category and each product in that category. A product page would have a tag that represents that product.

This allows Magento to selectively clear the Varnish FPC, say when a product’s content is modified, by sending a special header to varnish to PURGE pages that match the product’s tag.

This is a communication between Magento and Varnish.

(Magento is aware of the location of varnish by the “http_cache_hosts” setting, Varnish is aware of Magento backend IP in its acl_purge list).

Indexing

In order to understand how Magento triggers cache clear events, we need to understand indexing.

Magento has many indexes – a mechanism by which changes made to admin are reflected on the frontend (the end user UI). Varnish only needs to be cleared when Magento moves products to frontend.

If index modes are set to update on save, each product save will result in corresponding pages to be removed from varnish. When making large changes such as with a csv upload, this will result in many pages being cleared multiple times. Note that a product save results in both the product as well as all the category pages of the category / categories the product belongs to.

If indexing mode is set to on schedule, a cron task that processes the “index” group is responsible for running indexing. If acceptable to the business, indexing can be set to run every few hours, reducing the rate at which varnish clears.

If you want a no hassles managed hosting, get in touch with luroConnect. We even give you a free assessment of your website.

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.

Supercharge your Magento with a Varnish cluster

What is a varnish cluster?

A varnish cluster is a set of varnish nodes, each in a different geographical location, in front of the same Magento backend.

As shown a Magento hosted in the US East region can serve varnish nodes from across the world. Access to the website from each region is directed towards the nearest varnish, benefiting from lower latency and faster page loads.

If you serve customers in different regions - internationally or across the USA, your store can benefit from a varnish cluster.

Varnish and Magento performance

Magento 2 was architected to work with Varnish for improved performance. A typical webpage – category listing or product detail page - when returned from a cached varnish page (called a HIT in varnish) typically has a TTFB of a few milliseconds. An uncached page (also called a MISS in varnish) results in TTFB of a few seconds. Optimizing a MISS is very crucial, but we will not cover that in this article. We believe varnish is one portion of optimizing your website.

What is Network Latency

Latency is the time it takes for a network packet to go from your computer to the server with a request and come back with a response - assuming the response from the server was available immediately. When a page scores a HIT in varnish, the response is almost immediate from the server. Any  Time To First Byte (TTFB) recorded on the browser can be attributed to network latency.

If you market your website to many regions around the world, and you host varnish in a single location, your visitors may be faced with higher latency. When latency of access is in a few hundred milliseconds, it becomes the bottleneck and needs optimization.

The below information from https://wondernetwork.com/pings/ gives an idea of expected latencies. To read the table use the row header as source and column header as destination. So, a ping latency from Los Angeles to Mumbai would be 267 msec.

What is the cause of latency?

Core latency is a function of distance – even light will take ~40msec to travel 13,000 km – the distance from New York to Mumbai. A network packet travels through wires at that speed, and network wires are much longer than a straight line. Moreover, a response packet has to travel all the way back.

Latency is also caused as traffic goes through network equipment. The number of switches / servers a packet has to go through depends on the network and service providers – yours as well as at the server you are connecting to.

A varnish cluster architecture

Using a GeoIP based DNS service such as AWS Route53 a users request for a domain is redirected to one of several IPs. The Magento backend is in the “default” region – where maximum traffic is expected. A varnish in the regions desired – shown US East and Europe below.

Geo-IP based DNS

A geoip based dns router such as AWS route53 can help direct traffic to the nearest varnish node based on the guessing the country the IP requesting name resolution is coming from. So users on a browser say in Australia would be directed to be served from the varnish node in Australia and one from the west states of California would be directed to the varnish there.

Since IP to region or country can never be so accurate, it should be possible for all varnish nodes to serve customers from any region. Specifically, a language or currency switcher should be available on the website.

Magento 2 supports multiple varnish nodes out of the box

An important feature in using a cache in production is its need to automatically and quickly clear contents on demand from the user and application. For example, when a product goes out of stock, the corresponding page should display the out of stock label. Magento uses a tag-based system to flush appropriate content from the cache. Magento allows setting up multiple varnish hosts and a tag-based cache clear is sent to all the hosts.

bin/magento setup:config:set --http-cache-hosts=<varnish internal ip>:6081,<varnish internal ip 2>:6081

Refer : https://devdocs.magento.com/guides/v2.4/config-guide/varnish/use-multiple-varnish-cache.html



Challenges in a varnish cluster

A varnish cluster is more complex to manage

Ensure the varnish vcl files, front end security configuration (WAF, rate limiting, etc) is managed and kept in sync on all edge nodes.

Managing includes monitoring to ensure none of the servers go down. Now, your site can be down in a specific region for example. Typical monitoring tools such as Pingdom would not work. A purposeful monitoring solution is needed.

A varnish cluster costs for additional servers

Since these would be frontend servers, the amount of RAM and their network speed requirement would depend on what traffic they get.

Number of varnish nodes

Increasing the number of nodes in a varnish cluster does not always help in improving site speed. That is because each varnish node has a different hit ratio. A lower hit ratio leads to more users getting the latency and performance penalties combined – due to a varnish MISS. Traffic pattern and latency have to be taken into account to decide on how many nodes to use in a cluster.

Difficult to warm the cache

Given the distributed nature of the cache, warming each cache independently takes more resources on the server side as well as some changes to the way a cache warmer works.

luroConnect : A modern cloud hosting platform

Schedule a call of a free evaluation and demo!

  • Is horizontal scaling manually or with autoscale right for you?
  • Evaluate if a varnish cluster will help your website performance
  • Show managed dev, staging and production environments
  • How we measure application performance every minute with alerts for a slowdown
  • Can your hosting costs be optimized?
  • How improved hosting can lead to better ROI!

On Magento Cloud? We have special offers if you switch your enterprise license to luroConnect managed AWS cloud.


Magento Cloud and Fastly

It is important to discuss the Magento 2 cloud decision to use Fastly as a frontend.

 

Magento 2 cloud pro version architecture is given below. (Reference : https://devdocs.magento.com/cloud/architecture/pro-architecture.html)

The architecture uses Fastly as a Full Page Cache. Varnish is not installed on the Instances in AWS.

Fastly is a CDN that uses varnish. With the Magento 2 Cloud integration, a custom plugin is used on Magento along with a custom vcl file that runs on Fastly.

Fastly has many “POP” locations. As per fastly documentation, there are 20 POP locations in North America, https://www.fastly.com/network-map

mostly in the USA. Each has its own varnish cache. When a page is not in a POP, it fetches content from the origin. A single page may have to be rendered 20 times for each POP location in North America.

Drawbacks of this architecture from a varnish cluster perspective

  • More POPs do not lead to better experience as a higher percentage of MISS on varnish results in a worse experience for more users.
  • As POPs increase the load on Magento infrastructure increases.
  • It is not possible to use a cache warmer to warm the Fastly cache.

What next?

An ideal situation would be a layered varnish configuration - each "satellite" varnish node serving a local user, caching a subset of the "main" varnish node, reducing the penalty of a varnish MISS.

Share your thoughts here or on social media.