Toggle navigation OptaPlanner logo
  • Home
  • Download
  • Learn
    • Documentation
    • Videos
    • Slides
    • Training
    • Use cases
    • Compatibility
    • Testimonials and case studies
  • Get help
  • Source
  • Team
  • Services
  • Star
  • @OptaPlanner
  • Fb
Fork me on GitHub
  • Will Skynet control our schedule if the compute...
  • How much faster is Java 8?

Can MapReduce solve planning problems?

Mon 3 March 2014

Avatar Geoffrey De Smet

Geoffrey De Smet


Twitter LinkedIn GitHub

OptaPlanner lead

To solve a planning or optimization problem, some solvers tend to scale out poorly: As the problem has more variables and more constraints, they use a lot more RAM memory and CPU power. They can hit hardware memory limits at a few thousand variables and few million constraint matches. One way their users typically work around such hardware limits, is to use MapReduce. Let’s see what happens if we MapReduce a planning problem, such as the Traveling Salesman Problem.

About MapReduce

MapReduce is programming model which has proven to be very effective to run a query on big data. Generally speaking, it works like this:

  • The data is partitioned across multiple computer nodes.

  • A map function runs on every partition and returns a result.

  • A reduce function reduces 2 results into one result. Its continuously run until only a single result remains.

For example, suppose we need to find the most expensive invoice record in a data cluster:

  • The invoice records are partitioned across multiple computer nodes.

  • For each node, the map function extracts the most expensive invoice for that node.

  • The reduce function takes 2 invoices and returns the most expensive.

About the Traveling Salesman Problem

The Traveling Salesman Problem (TSP) is a very basic planning problem. Given a list of cities, find the shortest path to visit all cities.

For example, here’s a dataset with 68 cities and its optimal tour with a distance of 674:

mapReduceTspBlogExplanation

The search space of this small dataset has 68! (= 1096) combinations. That’s a lot.

A more realistic planning problem, such vehicle routing, has more constraints (both in number as in complexity), such as: vehicle capacity, vehicle type limitations, time windows, driver limits, etc.

MapReduce on TSP

Even though most solvers probably won’t go out of memory on only 68 variables, the small size of this problem allows us to visualize it clearly:

mapReduceTspBlog 0

Let’s apply MapReduce on it:

1) Partition: Divide the problem into n pieces

First, we take the problem and split it into n pieces. Usually, n is the number of computer nodes in our system. For visual reasons, we divide it into only 4 pieces:

mapReduceTspBlog 1

TSP is easily partitioned because of it only has 1 relevant constraint: find the shortest path.

In a more realistic planning problem, sane partitioning can be hard or even impossible. For example:

  • In capacitated vehicle routing, no 2 partitions should share the same vehicle. What if we have more partitions than vehicles?

  • In vehicle routing with time windows, each partition should have enough vehicle time to service each customer and drive to each location. Catch 22: How do we determine the drive time if we don’t know the vehicle routes yet?

It’s tempting to make false assumptions.

2) Map: Solve each piece separately

Solve each partition using a Solver:

mapReduceTspBlog 2

We get 4 pieces, each with a partial solution.

3) Reduce: Merge solution pieces

Merge the pieces together. To merge 2 pieces together, we remove an arc from each piece and add 2 arcs to connect cities of different pieces:

mapReduceTspBlog 3

We do merge several times until all pieces are merged:

mapReduceTspBlog 4

There are several ways to merge 2 pieces together. Here we try every combination and take the optimal one. For performance reasons, we might instead connect the 2 closest cities of different pieces with an arc, and then add a correcting arc on the other side (however long that may be).

In a more realistic planning problem, with more complex constraints, merging feasible partial solutions often results into an infeasible solution (with broken hard constraints). Smarter partitioning, which takes all the hard constraints into account, can sometimes solve this…​ at the expense of more broken soft constraints and a higher maintenance cost.

4) Result: What is the quality of the result?

Each piece was solved optimally. Pieces were merged optimally. But the result is not optimal:

mapReduceTspBlog 5

In fact, the results aren’t even near optimal, especially as we scale out with a MapReduce approach:

  • More variables result in a lower result quality.

  • More constraints result in a lower result quality, presuming it’s even possible to partition and reduce sanely.

  • More partitions result in a lower result quality.

Conclusion

MapReduce is great approach to handle a query problem (and presumably many other problems). But MapReduce is a terrible approach on a planning or optimization problem. Use the right tool for the job.

Note: We applied MapReduce on the planning problem, not on the optimization algorithm implementation in a Solver, for which it can make sense. For example, in a depth-first search algorithm, MapReduce can make sense to explore the search tree (although the search tree scales exponentially bad which dwarfs any gain of MapReduce).

To solve a big planning problem, use a Solver (such as OptaPlanner) that scales well in memory, so you don’t need to resort to partitioning at the expense of solution quality.


Comments Permalink
 tagged as tsp insight algorithm

Comments

Visit our forum to comment
  • Will Skynet control our schedule if the compute...
  • How much faster is Java 8?
Atom News feed
Don't want to miss a single blog post?
Follow us on
  • T
  • Fb
Blog archive
Latest release
  • 8.1.0.Final released
    Fri 15 January 2021
Upcoming events
  • KIE Live
    Worldwide - Tue 19 January 2021
    • OptaPlanner Shadow Variables for the Vehicle Routing Problem and Task Assignment by Geoffrey De Smet, Karina Varela, Alex Porcelli
  • Javaland
    Worldwide - Tue 16 March 2021
    • AI on Quarkus: I love it when an OptaPlan comes together by Geoffrey De Smet
Add event / Archive
Latest blog posts
  • Solve the facility location problem
    Fri 9 October 2020
     Jiří Locker
  • OptaPlanner Week 2020 recordings
    Mon 7 September 2020
     Geoffrey De Smet
  • Let’s OptaPlan your jBPM tasks (part 1) - Integrating the two worlds
    Fri 3 July 2020
     Walter Medvedeo
  • AI versus Covid-19: How Java helps nurses and doctors in this fight
    Fri 8 May 2020
     Christopher Chianelli
  • Workflow processes with AI scheduling
    Tue 5 May 2020
     Christopher Chianelli
  • Constraint Streams - Modern Java constraints without the Drools Rule Language
    Tue 7 April 2020
     Geoffrey De Smet
  • How to plan (and optimize) a Secret Santa
    Wed 18 December 2019
     Christopher Chianelli
Blog archive
Latest videos
  • YT Shadow variables
    Tue 19 January 2021
     Geoffrey De Smet
  • YT Domain modeling and design patterns
    Tue 17 November 2020
     Geoffrey De Smet
  • YT Quarkus insights: AI constraint solving
    Tue 20 October 2020
     Geoffrey De Smet
  • YT AI in kotlin
    Wed 23 September 2020
     Geoffrey De Smet
  • YT Planning agility: continuous planning, real-time planning and more
    Thu 3 September 2020
     Geoffrey De Smet
  • YT Quarkus and OptaPlanner: create a school timetable application
    Thu 3 September 2020
     Radovan Synek
  • YT Business use cases and the impact of OptaPlanner
    Thu 3 September 2020
     Satish Kale
Video archive

KIE projects

  • Drools rule engine
  • OptaPlanner constraint solver
  • jBPM workflow engine

Community

  • Blog
  • Get Help
  • Team
  • Governance
  • Academic research

Code

  • Build from source
  • Submit a bug
  • License (Apache-2.0)
  • Release notes
  • Upgrade recipes
Sponsored by
Red Hat
More coder content at
Red Hat Developers
© Copyright 2006-2021, Red Hat, Inc. or third-party contributors - Privacy statement - Terms of use - Website info